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Partnering to Prepare Rookie Teachers Online?* 


In June, 2013 the California Commission on Teacher Credentialing released a Program Sponsor Alert 
announcing that the Commission had adopted changes to Pre-service Preparation, Support, and 
Supervision requirements for the Multiple Subjects, Single Subjects, and Education Specialist intern 


programs in California (http://www.ctc.ca.gov/commission/agendas/2013-04/2013-04-agenda.html). 


As described in the Commission’s Program Sponsor Alert, preparation programs would now be required 
to provide pre-service teachers with content to support English Language Learner students (ELLs); 
specifically, novice teachers would need to be given access to the following: principles of educational 
equity and diversity, linguistic development, current research related to second language acquisition, 
classroom management, families and culture, forms of assessment specific to English language 
proficiency and development, differentiation, structured oral interaction, analysis of literacy demands, 
and specific pedagogies and strategies designed to support English language development. 


At almost the same moment, the New York State Education Department introduced a required and 
high stakes examination for novice teachers through which they would be able to demonstrate their 
capacity to “educate all students.” Given the bi-coastal need, the High Tech High Teacher Credentialing 
Program and New Visions for Public Schools, in partnership with the San Diego County Office of 
Education and the San Diego Unified School District, began a process to meet the change in new 
teacher preparation requirements. The partners made two decisions: first, they agreed to develop, 
deliver, and evaluate an online English Language Learner course to meet the new content 
requirements. Second, in preparation for course development, the partners designed and the New 
Visions team implemented an evaluation of the online and blended courses that had been developed 
for teachers in San Diego County who had sought to become highly qualified as defined by NCLB 
federal legislation (Trachtman, Koenigsberg, Zheng, Cho, & Marlow, 2014). 


The results from the San Diego County evaluation influenced the design of the first course for rookie 
teachers implemented in summer 2014. This 8-week course focused heavily on reading research on 
language acquisition theory, a focus designed to raise novice teachers’ awareness of the issues involved 
in teaching English Learners. Comments from participating New York and California novices that 
summer (e.g., “! now teach EL students and do not have resources from the course to help me because | 
did not get a solid grasp of how to apply the strategies shown in the videos or laid out in the readings.”) 
raised important questions for program designers. Through the collection of these anecdotal data 
coupled with a rigorous comprehensive evaluation completed by a team based at New Visions 
(Trachtman, Koenigsberg, & Zheng, 2014), six key areas were identified in need of improvement. An 
online education consultant worked with one of the course instructors to re-design the course for 
implementation in summer 2015. With additional philanthropic support, the partners evaluated the re- 
designed course. The study that follows reports on the new tools developed by the partners to 
evaluate their online courses and examines the results for both the 1*t and 2"4 course versions. 


Review of Related Literature: Online Learning and Teacher Development 


We begin this review of the literature with Palloff & Pratt’s (2007) caution: "It is always important to 
remember that in the online environment, we present ourselves in text. Because it is a flat medium, we 
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need to make an extra effort to humanize the environment. In the face-to-face classroom, students have 
the opportunity to get to know one another as people--before or after class, during classroom 
discussions, and in other campus locations such as the student lounge. In the online environment, we 
need to create these opportunities more purposefully." Many argue that online and blended courses are 
viable options for addressing the shortage of highly qualified teachers in hard-to-staff subjects, in part 
because of the flexibility they afford to both students and instructors. Smith, Smith, & Boone (2000) 
sought to compare the efficacy of online and traditional face-to-face instruction; the study’s sample was 
comprised of nearly 60 preservice educators enrolled in a course on “educational technology 
integration.” Results showed that student outcomes in online and face-to-face instruction were 
comparable, although the authors posited that online discussion provided an advantage over its 
traditional counterpart. Esprivalo Harrell, & Harris (2006) compared the online post-baccalaureate 
teacher education program at the University of North Texas to the institution’s traditional baccalaureate 
teacher education program and found that the online program was more successful in attracting a 
diverse candidate pool, including minority students and career changers. The online program also 
attracted more students to teach math and science, both subjects identified as “critical shortage areas.” 
The authors also report that students in the online program had comparable teacher quality ratings, as 
determined by GRE scores, state certification exams, and portfolio ratings, and that students were 
generally satisfied with the online program. 


Denton, Davis, Capraro, Smith, Beason, Graham, & Strader (2009) sought to determine what 
biographical and academic characteristics could best predict whether an applicant to an online teacher 
education program would enroll and complete the program. They included three years of program data, 
including 170 applicants and found that the characteristic most indicative of success in the program was 
the applicant’s TExES score (the statewide praxis exam in Texas). Huss (2007) sought to understand 
principals’ feelings toward online teacher preparation programs. By analyzing the survey responses of 
326 principals across three states selected via random cluster sampling, the author determined that 
main areas of concern for principals included how social aspects of teaching might be compromised in 
preparation programs that were online. A majority (59%) of principals involved in the study reported 
that they would be “very concerned” if candidates for teaching positions reported that their degrees 
were primarily attained online; likewise, 95% of principals felt that an online degree did not have as 
much credibility as one obtained via traditional pathways (p. 3-4). Nevertheless, as early as 2005, Olson 
and Werhan suggested that online coursework was the next frontier of teacher preparation, especially 
in traditionally hard to staff subjects. They developed their hypothesis in the context of the rise in 
alternative teacher certification programs, and the need for remote school districts to meet NCLB’s 
requirement related to highly qualified teachers. 


Fisher, Schumaker, Culbertson, & Deshler (2010) compared the effects of computer-based professional 
development with those of face-to-face professional development using data from two studies. The first 
found that that teacher’ scores on both knowledge and concept diagram tests improved comparably 
after participating in professional development in either medium, suggesting equitable efficacy. It is, 
however, worth noting, that teachers who participated in the face-to-face program expressed greater 
satisfaction that the authors attributed to engagement. The second study focused on implementation 
and found that participants in both types of professional development programs continued to use the 
instructional behaviors emphasized in the training. 


Smith et al. (2000) posited that online and distance learning offered viable solutions to both the current 
lack of certified special education teachers and the need for relevant and progressive professional 
development for teachers currently serving students with special needs. Likewise, Salazar and 


colleagues (2010) highlighted the merits of online professional development, including widespread 
access and increased flexibility for participants. They also suggested that online learning allowed for a 
freer exchange of dialogue. From an administrative perspective, they noted that online professional 
development courses were cost-effective and allow for cost-sharing arrangements. 


Chen, Chen, & Tsai (2009) analyzed transcripts of six online teacher professional development sessions 
(containing a total of 3600 messages), and interviewed some of the participants. The authors found that 
online discussions served several purposes beyond continued learning, including as a venue for 
networking and socialization. Their analysis found that over half of the messages were not related to the 
topic at hand (p. 1163), although some of these messages were used for teachers to exchange ideas 
about practice. An additional 27% of messages were designated as “social cues,” leaving just about 28% 
of messages that researchers established as involving “cognitive of meta-cognitive” skills. Authors 
suggest that a scaffolded model with more instructor participation could enhance online professional 
development courses. After completing interviews with participants, the authors suggested that 
synchronous discussions in the context of online professional development may not have a significant 
advantage over face-to-face conversations. 


Key findings from Dede et al. (2005) pointed out the need for deliberate structures to facilitate 
participant engagement, arguing that social presence was related to student perceptions of learning and 
course Satisfaction, and that online professional development programs provided a less interactive but 
more reflective environment than face-to-face programs. Dede et al. (2009) stressed that much of the 
available literature was anecdotal or descriptive, and cited the need for the development of an 
evidenced-based framework moving forward. 


Literature on designing blended learning programs, either as professional development programs 
themselves or to disseminate blended learning strategies to practicing teachers, reflects the ebb and 
flow of interest from the field. Prior research focused on the effects of technology-based professional 
development on praxis, which found that “participation in research-based PD programs fosters 
sustained changes in teachers’ instructional technology knowledge, ability to design and implement 
technology-enhanced learning experiences for students, and positive attitude toward teaching and 
learning with technology, (Mouza, 2009, cited in Duran et al. 2012, p.316).” Duran et al. (2012) also 
found that the wiki-based professional development program they studied was particularly successful 
because of the broad application of the technology in question and support at the district level for the 
program. Wach, Boughton, & Powers (2011) found that faculty peer mentoring practices and the 
presence of student instructional technology tutors supported the development of hybrid courses at 
Bronx Community College. 


In its guidelines for teaching online, the Southern Regional Education Board (2009) recommended that 
faculty engage students through the use of differentiated instruction, supplemental materials, and a 
variety of technology and respect the differentiated needs of learners. Similarly, Baran et a/. (2011) 
advocated for the inclusion of learner-centric approaches, as students needed to take greater 
responsibility and a more active role in their learning in an online or blended course (Baran et al., 2011, 
p. 429). Baran et al. (2011) also stressed the importance of involving teachers in all facets of the design 
of blended learning courses: “Support programs need to consider teachers as active agents during this 
process. Instead of building courses for them, a collaborative culture around course design and 
development needs to be provided and supported. Technology staff and instructional designers should 
constantly engage in a dialogue about solving problems and making decisions regarding the design and 
teaching processes of online courses” (p. 435). 


Jokinen & Mikkonen (2013) found that collaborative planning was helpful for teachers of blended 
learning courses despite the time it required. In general, teachers reported that engaging in the 
development and teaching of blended learning courses allowed them to integrate and, at times, 
condense courses and subjects, and that they encouraged teachers to try more creative and effective 
methods. De Gagne and Walters (2009) focused on the experiences of instructors of online courses in 
their 2009 meta-analysis. They noted that extra preparation and organization was required of online 
instructors, due in part to the constant flow of feedback from participants and the LMS-collected data. 
Instructors also found themselves transitioning from lecturers to guides and facilitators of engagement; 
teaching strategies were affected both by prior technological experience and the ability to learn about 
their students’ instructional needs before the course started; and opportunities for professional growth 
are inherent in online teaching opportunities but often require additional administrative support. The 
authors conclude that social presence and a focus on learner-centric practices are required in teaching 
online courses. Additionally, they noted teachers of online courses required administrative support, 
training, and performance-based incentives. 


Several studies argue that online components of blended learning courses should supplement face-to- 
face instruction, not replace it, and should be introduced gradually to ease the transition for teachers to 
implement blended learning systems within their curricula (University of New South Wales; Means et 
al., 2013). Application of blended learning initiatives also requires a significant time commitment at the 
beginning of the project. Napier, Dekhane, & Smith (2011) highlight some of the challenges associated 
with transitioning to blended learning, including time management, balancing course components, 
engaging students with the material, and providing enough support to students outside of the 
classroom--especially in regard to determining whether students have the self-discipline required for 
success in a course with online elements. 


Much of the literature emphasized the importance of promoting group work and developing a learning 
community in the classroom (SREB 2009), in addition to promoting communication among students 
outside of class time (Napier et a/., 2011). Students should also be provided with opportunities to 
become acclimated to the online environment and/or software participation that the blended course 
requires. Student interactions and community building are also the focus of McElrath & McDowell’s 
(2008) work. They offer strategies for community creation in online courses, noting that previous 
research shows that students in online courses often elevate the role of the instructor to that of a model 
and facilitator of community building. The authors cite Ruth E. Brown’s 3 stage process of community 
building in distance learning (making friends online, community conferment, and camaraderie) and offer 
strategies to enable student success, ranging from including a chat function to using student stories to 
illustrate frameworks. Also important in a blended learning context is attentiveness to student 
responses and timely feedback (SREB). The SREB also emphasizes that students in blended learning 
classes should understand “digital etiquette” and have a respect for intellectual property and other 
copyright concerns associated with digital content. 


Mandernach, Gonzales, & Garrett (2006) also focused on online course instructors, noting that teachers 
of such courses could be assessed in the same manner as teachers of traditional courses. They highlight 
instructor presence--specifically citing teaching presence, instructor immediacy, and social presence--as 
key capacities for online course instructors. The authors cite Savery’s (2005) model for online teacher 
success-- (2005) presents the following model for online teacher success: Visible, Organized, 
Compassionate, Analytical, Leader-by-example. Their study involved surveying 96 experienced online 
course instructors; in concluding, they write: “Faculty perceptions regarding the regulation and 


evaluation of instructors’ participation in online course threaded discussions suggested that specific 
benchmarks regarding time and frequency of instructor posting in the online classroom may not be as 
important as setting professional expectations and communicating concrete strategies for instructors’ 
visibility in the online classroom.” 


Babb, Stewart, & Johnson (2010) surveyed 75 undergraduate students enrolled in hybrid courses. 
Respondents identified the following as best practices: user-friendly website, well-designed 
assignments, a forum for faculty/student communication, building a community of learners, prompt 
feedback mechanisms, and communicating expectations. Establishing benchmarks was also strongly 
recommended, as useful “in predicting student learning outcomes and satisfaction.” A forum for student 
discussion is also recommended, as the authors found there to be a positive relationship between 
students’ satisfaction and their interaction with their peers. 


Taylor’s (2011) faculty competencies for online teaching included the following: 


Attend to unique challenges of distance learning 
Be familiar with unique learning needs 

Master course content, structure and organization 
Respond to student inquiries 

Provide detailed feedback 

Communicate effectively 

Promote a safe learning environment 

Monitor student progress 

. Communicate course goals 

10. Provide evidence of teaching presence 


WONDNAWNE 


In their description of online teaching principles and online teaching behaviors, Tobin, Mandernach, 
Taylor, (2015, p. 129) recommended these specifics: 


Encourage student-faculty contact 

e Set aside regularly scheduled times for online office hours or implement a maximum turnaround 
time for responses to communications 

e Facilitate regular course discussions 

e Post course announcements or news updates on a regular basis 


Develop reciprocity and cooperation among students 

e Assign group or dyad projects 

e@ Require discussion responses to peers. 

e Offer encouragement in public ways (e.g., on the course discussion forum); offer criticism privately 
(n grade-tool feedback seen only by individual students). 


Use active learning techniques 

e Ask students to summarize and propose next steps. 

e Assign “butts out of seats” tasks to give online learners tasks away from the keyboard (e.g., 
interview experts near students’ homes) and ask students to report back to other class. 

e Have students create and post study guides. 


Respect diverse talents and ways of learning 


e Provide multiple ways for students to respond to assignments (e.g., write an essay, recode an audio 
response, create a video) 

e Allow students to respond to discussions using a variety of media 

e@ Present learning material in a manner that enables a range of possible learning paths. 


Chickering & Ehrmann (1996) and Chickering and Gamson (1987) Principles of Good Practice 


Encourage student-faculty contact 

Develop reciprocity and cooperation among students 
Use active learning techniques 

Give prompt feedback 

Emphasize time on task 

Communicate high expectations 

Respect diverse talents and ways of learning 


SEO oe ON 


Changing Roles, Relationships, and Competencies. While much has been written about the learning 
strategies, staffing models, processes, and products of student-centered blended learning, the glaring 
gap in the documentation and research — and a key driver to transforming any learning system — has 
been a focus on the teacher. The new definition of student success requires new definitions of teacher 
and teaching success. In these classrooms, the role of the teacher is both essential and fundamentally 
different. While many have expressed fears that the introduction of technology and open access to 
online content displaces the centrality of teachers, student centered tech-enabled learning provides 
teachers with a clear and direct pathway to personalize learning to meet students’ needs. 


Instead of standing at the center of the classroom as the repositories of knowledge and the gatekeepers 
of student learning, teachers stand at the student’s shoulder, coordinating students’ learning as guides 
and facilitators. Instead of whole class instruction in front of an undifferentiated mass of learners, 
teachers customize instruction to the needs of students, based on analysis of real-time data. Internet 
access and digital tools give teachers and students a larger learning environment and access to 
knowledge that connects them to people and resources beyond the classroom. With more space for 
individualizing, teachers have new opportunities to build nuanced and more nurturing relationships with 
students as they design learning experiences that help students develop their assets and take 
responsibility for their own growth Williamson, Shaffer, Nash, & Ruis, 2015). 2 


Teaching presence is the mechanism that bridges the transactional distance between instructor and 
student in a virtual classroom where direct instruction and facilitation of discourse are achieved through 
various forms of interaction (Afolabi, 2016; Arbaugh & Hwang, 2006). Ekmekci (2013) and Bowden 
(2012) presented arguments of the responsibilities instructors hold in setting academic expectations and 
ensuring that standards of scholarship are upheld. Instructors can create an academic climate that 
increases connectedness with students and expectation of scholarship by promoting a shared sense of 
teaching presence (Afolabi, 2016; Ekmekci, 2013). This sense of “being there” or “being together” is 
experienced in different ways in the online classroom and must be intentionally created for it to be 
perceived and felt (Lehman & Conceicdo, 2010). The importance of teaching presence and its influence 
on the learning process has increased in visibility (Arbaugh, 2013). However, as the online teaching 
environment has advanced, instructors have not always kept pace, showing a tendency to either rely too 


2 Shaffer, Nash, & Ruis. (2015). Technology and the new professionalization of teaching. Teachers College Record, 117(12), 1- 
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heavily on technology to form connections with their students or to revert to conventional practices that 
are more suitable for physical classrooms (Baran et al., 2013; Cho & Kim, 2013; March & Lee, 2016). 
Those who are new to online environments are challenged with finding suitable approaches to teaching 
in virtual classrooms because many did not learn that way themselves (Niess & Gillow-Wiles, 2013). 


The Evaluation Design 
Evaluation Questions and Design 


1. What is the overall quality of the online modules? 

2. What is the impact of the professional development on teacher preparedness, attitudes, and beliefs 
about teaching and learning? 

3. How does participation in this program affect novice teachers’ perceptions of efficacy? 

4. How do the overall quality and impact of the EL Phase 1 (2014) course compare to that of the 
redesigned EL Phase 2 (2015) course? 


Mixed Methods Design, Instrumentation, and Limitations.? The instruments created for this evaluation 
included a set of rubrics (Appendix A) for evaluating course videos, discussion boards, and assignments 
and an online survey (Appendix B). Survey development included a review of the online literature and 
items from the teacher professional development and efficacy research. 


Since we were not able to locate earlier empirical studies evaluating online instructional videos, 
discussion board dialogues, and student work produced during novice teachers’ participation in online 
professional development, we used a more exploratory approach to this component of data collection 
and analysis. Our researcher-developed rubrics (see Trachtman, Koenigsberg, Zheng, Cho, & Marlow, 
2014), based on well-accepted theoretical foundations, were tested for face validity with a small group 
of experts. Additional support for the rubrics’ validity was provided by the results of three prior 
evaluations where rubric ratings generally corresponded appropriately with survey results. In terms of 
reliability, we attempted to limit measurement error by having only one researcher review all artifacts 
from the novice teachers’ online experiences. Although circumstances dictated that the researcher 
responsible for rating the course videos, discussion board dialogues, and student work in the Phase 1 
evaluation was not able to work on the second evaluation, we were fortunate in that she was available 
to train her replacement. The original researcher worked with the new analyst to calibrate their scoring 
so that artifacts would be rated consistently. After they reviewed and discussed the dimensions of the 
rubrics together, the new analyst independently scored the artifacts that had been rated in Phase 1 and 
produced her own ratings. When these differed from the initial ratings, she discussed the reasons for 
the discrepancy with the original rater. This norming process continued until a high level of agreement 
was reached. The lead evaluator who reviewed the new researcher’s comments provided an additional 
check on their accuracy. 


Survey Components and Constructs 


Course Experience. Items for the Course Experience section of the survey replicated the survey items 
from the previous set of studies completed by the evaluators (see Trachtman et a/., 2014). The 


3A post hoc analysis showed that the Instructor Feedback rubric contained a criterion conflating frequency and quality 
which raised some concern about the resulting scores. 


evaluators and VPSS staff had selected these items from the Student Perceptions of Course Delivery 
Survey (Babb et a/., 2010). Based on a factor analysis of the items, three course experience scales were 
constructed: Instructor Engagement, Culture of Learning, and Course Design. 


Course Components Effectiveness. For each of the VPSS and EL evaluations, items were developed after 
identifying the most important components in the respective course syllabus. Changes in the 
component list for the different evaluations reflect modifications in the syllabi. Half of the 2015 
components kept the same name as in 2014 and half were renamed. Although they were redesigned 
and new content was added, Readings, Videos, Discussion Boards, and Presentation of Learning were 
similar enough in function to retain their 2014 names. However, the Individual Wiki Projects in 2014 
became Partner Team WikiProjects in 2015, reflecting a fundamental change in the content and 
structure of the component. Similarly, Group Work (2014) was changed to Partner Teams in 2015, 
Reflections (2014) was renamed Module Reflections, and Prior Knowledge Assignments (2014) was 
changed to Prior Knowledge Thinking Prompts in 2015. Using the same strategy as in earlier evaluations, 
we created a matrix for respondents to score the effectiveness of each of these course components in 
relation to the three major course goals. 


Teaching Preparation. Imbimbo and Silvernail (1999) created a set of items to examine novice teachers’ 
beliefs about their own abilities. Nineteen of the Imbimbo/Silvernail items were selected for the VPSS 
survey by VPSS personnel and used again in both the 2014 and 2015 EL surveys. 


Gibson and Dembo Teacher Efficacy Instrument. Gibson and Dembo (1984) developed their 2-factor 
instrument from Bandura’s social cognitive theory and his construct of self-efficacy (1977, 1982). The 
first factor, personal teacher efficacy (PTE), includes 16 items that tap teachers’ beliefs related to their 
ability to bring about student learning. The second, called general teacher efficacy (GTE), includes 9 
items that tap teachers’ beliefs about the ways in which external factors, such as the home 
environment, family background, and parental influences, limit the teacher’s ability to affect student 
achievement. Figure 2 in Appendix E presents the PTE and GTE items. 


Design and Assessment of Module Artifacts 


Videos. Instructional videos were evaluated with a rubric developed from the Danielson teacher 
evaluation framework and educational research on the effective use of instructional videos in the 
classroom (Canning-Wilson & Wallace 2000; Cruse 2011; Danielson 2013). The rubric domains included 
designing coherent instruction, establishing a culture for learning, and engaging students in learning. 


Discussion Boards. Discussion boards were evaluated with a rubric developed from criteria constructed 
by TeacherStream (2009). Domains included Instructional Feedback, Demonstration of Knowledge of 
Key Concepts, Community Building, Reflection, and Critical Thinking. 


Instructor Assignments and Student Work. Instructor assignments and associated student work products 
were evaluated with a rubric developed from the teacher practices and student work indicators in the 
Danielson Framework (2013). Domains included assignment design, student response to assignment, 
and instructor response to student work. 


Findings 


In order to maximize the mixed methods data collection strategy, we have integrated the presentation 
and analysis of findings from the review of course artifacts (videos, discussion boards, assignments) and 
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the survey. Each source of data, however, uniquely contributes to providing answers to the original 
research questions. Accordingly, the following sections present: a) the observations and discussions 
based on the examination of artifacts, b) the results of the analyses of survey data, and c) a discussion of 
the survey results, both separately and with regard to the course artifact analyses. 


Video, Discussion Board, and Assignment Findings 
Instructional Videos 


The Sample. \|n our initial examination of the EL course modules, we determined that the videos fell into 
the three instructional categories identified last year: 


1. Pedagogical Theory and Strategy. Videos that discuss classroom-related instructional practices (e.g., 
an animated introduction to the “plan and let go” approach) or specific strategies (e.g., a lecture on 
welcoming new EL students) 

2. Cultural and Historical Context. Videos that are not solely geared to educators and provide 
information about EL culture and history (e.g., a description of second language acquisition) 

3. Teaching/Classroom Modeling. Videos that show teachers in classrooms modeling instructional 
practices (e.g., footage of a teacher demonstrating TPR [total physical response] techniques in the 
classroom) 


We randomly sampled 12 instructional videos, maintaining the same ratios used in the first evaluation’s 
sampling process. We included nine Pedagogical Theory and Strategy videos; one Cultural and Historical 
Context video; and two Teaching/ Classroom Modeling videos. The evaluation rubric for the 
instructional videos was derived from the 2013 Danielson framework and educational research on the 
effective use of instructional videos in the classroom (Canning-Wilson & Wallace 2000; Cruse 2011; 
Danielson 2013). After determining the relevant Danielson domains, we established 3 broad assessment 
criteria, constructed multiple indicators, and assessed each on a scale of 1-5, with 5 as the highest score. 
We then determined a total criterion score by averaging all individual indicator scores (see Appendix A). 


2015 Video Findings 


e The Teaching/Classroom Modeling videos were strong across all three criteria, earning average 
scores of 5 for Designing Coherent Instruction, 4.9 for Engaging Viewers in Content, and 5 for Video 
Production and Viewing. 

e Some of the highest scoring Teaching/Classroom Modeling videos also included elements of 
Pedagogical Theory; likewise, some of the most successful Pedagogical Theory Videos were enriched 
by examples of Teaching/Classroom Modeling. An example of this hybrid form is the video “Best 
Practices: Checking for Understanding.” 

e Video scores were generally high across all indicators, with the exception of one: “Video shows 
pictures or visuals directly related to the information or lesson being presented.” Teaching/ 
Classroom Modeling videos scored well on this indicator (average score of 4.5), because, by 
definition, they showed footage of students and teachers interacting in the classroom. Pedagogical 
Theory videos received an average score of 3.6 on this indicator; while some of them used lively 
animated illustrations or classroom scenes to bring their content to life, others included no visuals 
(and showed only the speaker’s face). Finally, the Cultural and Historical Context Video received a 
score of 1 on this indicator, as it showed no illustrations, only text summaries of what the voiceover 
narration was discussing. 
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e The Cultural and Historical Context video generally received lower scores than the 
Teaching/Classroom Modeling and Pedagogical Theory and Strategy videos. Although the content 
was related to the module and the speaker did attempt to apply the material by including some 
classroom examples, the concepts were complex and theoretical, and therefore less accessible to 
novice teachers. Additionally, the video was more than 16 minutes long and not broken up into 
sections, and, as mentioned above, contained no visuals related to its topic; this resulted in a score 
of 3.7 in Engaging Viewers in Content and a score of 3 in Video Production and Viewing. 


2015 Video Findings: Discussion 


The most successful videos were Teaching/Classroom Modeling videos and Pedagogical Theory and 
Strategy videos that integrated aspects of classroom modeling and aspects of pedagogical theory. While 
modeling and discussion of pedagogy are both valuable learning tools on their own, it appeared as 
though they were more powerful when combined. 


Visual presentation emerged as a highly significant factor in the success of a video. From Teaching/ 
Classroom Modeling videos that showed an instructor demonstrating a technique with her students to 
animated Pedagogical Theory and Strategy videos that showed colorful cartoon representations of what 
teaching strategies look like in action, the highest scoring videos found a way to bring their content to 
life with visuals. 


A significant addition to the 2015 curriculum was a series of introductory videos. These videos were 
placed at the beginning of selected modules in the first half of the course (Modules 2, 3, and 4). Each 
informal and friendly introductory video was approximately one minute long, and served to welcome 
students to the module and prepare them to navigate its structure and content. 


In addition to orienting students to the modules, the introductory videos also made the experience of 
online learning more personal, in part by preparing students to upload their own videos. The first 
introductory video (in Module 2) told students about a WikiProject called the “All About Me video.” By 
presenting the video WikiProject assignment in the context of her personal video, the instructor 
modeled the format and gave the students something to respond to as they created their own videos. 
Participation rates were high; 100% of students in each class submitted All About Me videos, and these 
videos were received with thoughtful and welcoming remarks in the form of comments from instructors 
and fellow students. 


Comparison of 2015 Videos to 2014 Videos 


e Intwo out of three categories, the 2015 videos scored higher than the 2014 videos: 4.7 (up from 3.9 
in 2014) for Designing Coherent Instruction, and 4.4 (up from 3.6 in 2014) for Engaging Viewers in 
Content. The Video Production and Viewing score was slightly lower (4.1, down from 4.4), due to 
low scores on the “Video shows pictures or visuals directly related to the information or lesson 
being presented” indicator. Many videos in the Pedagogical Theory and Practice and 
Cultural/Historical Context categories showed only the lecturer’s face as they spoke or only textual 
summaries of what the lecturer was saying, without any illustrations or video footage of what they 
were describing. 

e The 2014 videos received low scores on two indicators, “Video demonstrates the how-to and how- 
not-to of the lesson” and “Video is appropriate to the learning needs of novice teachers.” 2015 
videos were more successful in these areas, by providing more deeply reflective commentary that 
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touched upon both what to do and what to avoid when implementing teaching practices, and by 
applying theories to classroom scenarios instead of simply presenting them as abstract ideas. 

e Although it received lower scores than 2015 videos in other categories, the 2015 Cultural and 
Historical Context video was stronger than its 2014 counterpart. The 2015 Cultural and Historical 
Context video received a score of 4 for Designing Coherent Instruction (up from 3 in 2014) and a 3.7 
in Engaging Viewers in Understanding Content (up from 2.5 in 2014). Even though its attempts to 
make the material accessible to novice teachers were not entirely successful, the fact that it did 
apply its ideas to classroom scenarios put it ahead of 2014’s Cultural and Historical Context video, 
which failed to do so. 

e Only seven videos from 2014 were reused in the 2015 curriculum, but some of the videos that 
weren’t featured were still available for student viewing in a section of optional learning materials. 
Each module in the 2015 course included a non-mandatory section called “Do You Want to Deepen 
Your Understanding?” that contained these videos and other optional articles and links. 


Comparison of 2015 Videos to 2014 Videos: Discussion 


While some video content remained the same, many revisions were made around it. For example, even 
though the revised 2015 version of Module 2 uses the same three videos as the original 2014 version, 
the theme of the module has changed (from “Responses to Issues of Educational Equity for ELs” to 
“Getting to Know You”) and the accompanying activities have changed accordingly. In 2014, the project 
team chose to use the same three videos to support a theory-based curriculum in which students were 
asked to write a Philosophy of Education based on the concepts they were learning, while in 2015, the 
same videos were used to support a more hands-on approach that asked students to create a family 
history and cultural backstory for a fictional student and plan a learning experience to meet that 
student’s needs. 


The overall move by the designers to create a more learner-engaged pedagogy can also be seen in the 
revisions to Module 1. In 2014, Module 1 introduced the course with three Cultural/Historical Context 
videos. In 2015, these videos were replaced with four Pedagogical Theory and Strategy videos. Instead 
of starting students off with background information about the history of education (e.g., “Brown vs. 
The Board of Education”) as they did in 2014, the project team decided to begin the 2015 course by 
immersing students right away in the how-to of teaching, with videos about pedagogical strategies and 
their applications. Note that three of these four pedagogically focused videos were repeated from 2014; 
however, in 2014 they appeared toward the end of the course, in Module 7. The 2015 course brings 
them to the front. 


In general, the 2015 videos were more successful in presenting learners with opportunities to 
understand how to apply lesson content. While some 2014 Pedagogical Theory and Strategy videos 
offered the novices broad and abstract theories, and other 2014 Teaching/Classroom Modeling videos 
lacked sufficient instructor reflection, the 2015 videos showed improvement on these indicators (“Video 
is Appropriate to the Learning Needs of Novice Teachers” and “Video Demonstrates the How-To and 
How-Not-To of the Lesson”) and generally reflected the instructors’ strong commitment to putting 
theory into context and making the material accessible. 


Discussion Boards 


The Sample. We chose to assess discussion forum prompts from the same modules as those reviewed in 
the first evaluation as well as one additional prompt from Module 7. While the 2014 modules each 
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offered a single prompt, in 2015 each module offered two prompt options for participants. In response 
to this change in the discussion board structure and student postings, our first analysis focused on 
calculating the number and percentage of students who chose each prompt option to determine what 
we are calling the “preferred prompt” for each module. In addition to assessing the preferred prompts, 
we assessed two non-preferred prompts in order to examine whether a certain kind of prompt elicited 
more responses. No prompts from 2014 were used again in 2015. 


Class 1 and Class 2 students preferred the same discussion forum prompts across all modules. However, 
we decided to report on each class’s forum scores separately because the instructors’ and participants’ 
postings and responses were distinct. 


The Framework. We utilized the 2014 discussion forum evaluation rubric again. As described previously 
(Trachtman et al., 2014), the rubric was modified from criteria initially created by TeacherStream 
(TeacherStream 2009). Each forum was assessed on eight indicators across five criteria and rated on a 
scale of 1-5, with 5 representing the highest score. The average score for each criterion was calculated 
afterwards (see Appendix A). 


We also derived a student participation percentage based on the number of discussion threads started 
by participants compared to the total number of participants enrolled in the thread. A 100% 
participation percentage meant each participant created at least one new discussion thread in the 
forum. The participation frequency of the course instructor was determined from the number of 
instructor replies to participants’ discussion threads. 


Choosing a Prompt: Findings 


e Students tend to prefer engaging with prompts that are simple and accessible. For example, both 
classes showed a preference (78% of class 1 and 60% of class 2) for the module 5 prompt that asked 
them to describe a project and its potential problems and possible solutions over the alternate 
prompt in the same module, which asked them to invent a project that could connect a “big idea” to 
a Common Core State Standard. It is possible that students find prompts that use education jargon 
(“big idea,” “Common Core”) less accessible than prompts written in simpler language (“projects,” 
“problems”). 

e Students seem to show a preference for answering questions that ask about their personal 
experience. In module 4, 100% of students in class 2 and 89% of students in class 1 chose to answer 
a prompt that asked them to describe a time in their lives when their affective filter came into play 
when learning a new language; the other prompt asked students to talk about their opinion of 
CELDT scoring. 

e In module 2, students showed a slight preference for the second prompt; 61% of class 1 students 
selected this prompt as did 55% of class 2 students. Generally, students in both classes made similar 
prompt selections. 

e Inboth class 1 and class 2, some students participated in both prompts, by posting an initial 
response in one and then replying to other students’ responses to the alternate prompt.‘ Out of the 
12 discussions we sampled, 11 included two or more comments from students who primarily 
responded to the alternate prompt. 


Prompt Selection: Discussion 


4 Note that participation rates included only posts by students enrolled in a given thread. 
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Prompt options were a new addition to the curriculum in 2015. Instead of replying to a single 
predetermined prompt as they did in 2014, 2015’s students were instead invited to make a choice, and 
fulfill their discussion requirement by addressing the prompt to which they felt most drawn. Throughout 
the course (and particularly in modules 1 and 5), readings and videos present arguments for presenting 
P-12 students with a choice about what they learn or how they learn in order to help make them feel 
more engaged in and excited about their work. Indeed, we see that students in both classes are actively 
engaged with their chosen discussion forums; the participation rate is 100% in 7 out of 8 forums 
examined, and both classes have consistently high Knowledge of Key Concepts (4.3 average) and 
Reflection (4.4 average) scores. 


In the preferred prompts especially, students worked together to build ideas and reach consensus. For 
example, module 2’s preferred prompt asked students about which teaching techniques would be most 
and least effective for their EL student. Students replied to each other’s posts not only by simply praising 
their ideas, but also by suggesting ways to modify techniques and sharing relevant teaching experiences 
of their own. For example, one student responds: 


“(Student 1], | too had problems with the action thermometer. And often find 
myself in the ‘middle’ of two extremes. | would alter the thermometer to offer 
more choices and diversity of response. Showing children that there is always 
more than two choices is an important example to set early on. And encouraging 
students to search for more alternatives on their own, is great practice in active 
problem solving.” 


This kind of creative, critical feedback kept the conversation deeply focused on one of the module’s 
learning objectives: tailoring learning experiences to students’ cultural backgrounds and personal needs. 


As they did in module 2, students sometimes drew from their classroom experience to enrich their 
contributions to the conversation across the discussion forums. In the preferred module 7 prompt, 
students were asked to describe potential pitfalls in providing feedback to students and talk about how 
to avoid them, and in their replies, some students talked about relevant ideas they’ve learned from their 
teaching experience: 


“| really resonated with the speaking with students one to one. This is 
something that | really try and do with my class. Actually, this is the only reason 
| have kept my teacher desk. | needed a place that was away from the groups of 
students, where | could converse with individuals without my comments or 
theirs being broadcast to the rest of the class. | believe it builds trust and 
creates a safe environment.” 


“You bring up a great point about timeliness and multiple perspectives in 
feedback. Sometimes, we as teachers can give a piece of feedback that doesn't 
register with a student. But when another person, whether it be a peer or 
parent, gives them the same feedback, it registers! | find that all the time with 
my students and as a coach. | also agree that giving feedback at the end can 
inhibit us from being engaged in the process, which translates to our students.” 
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In some cases, it appears that the nature of a discussion question helped to shape the quality and type 
of student responses. For instance, the preferred prompt in module 5 asked students to talk about an 
idea for a project, list potential problems with the project, and suggest a solution for one of the 
problems. Because the resulting conversation consisted of critiquing and making suggestions for 
improving student project ideas, it was nearly impossible for students to participate with passive “back- 
patting”; i.e., by simply praising other students’ posts. In order to engage with the prompt, students 
were required to acknowledge problems with their peer’s ideas and consider solutions. Because the 
prompt demanded a critical eye, student participation received high scores in Community Building (4.5) 
and critical Thinking (4). See Discussion Forum Scoring Chart. 


“Cool project! The first thing that jumped into my mind while reading through 
your project was that a student could run into a figurative brick wall if they 
picked a mathematician whose work was too complex. | completely agree with 
your solution, and might also suggest that you could pick a set pool of 
mathematicians whose work fits within the right criteria (not too simple, but 
not too complex for your students to handle). This would still give them 
freedom of choice, but without the demoralizing effect if a student 
inadvertently picked something too difficult.” 


One practice that seemed to help build active, productive discussions was a pattern of participants 
including questions in their posts. When students asked questions, the instructor was quick to respond, 
and often replied with questions of her own. For example, look at the questions asked by a student and 
the instructor in separate threads of the preferred prompt discussion in module 4: 


[Student] “I realized that my filter comes and goes depending on the 
environment. | too, thought about how to create a positive classroom 
environment when reflecting on my experience with an affective filter. How, 
though, can we make our own classrooms a space where students can feel 
highly motivated, self confident, and without anxiety? | feel that throughout 
this course | keep saying that | am going to create this welcoming, 
comfortable, safe space, but | am still unsure of how to do this exactly.” 


[Instructor] “Yes, great examples of how a positive experience and excitement 
around the process of language learning can encourage language production 
and advancement. Thanks for sharing. I'm wondering what you think a 
teacher might do to create a similar feeling for students who, for example, 
don't come in with a similar excitement?” 


These questions led to conversations that linked the personal affective filter experiences that the 
prompt asked students to share with bigger-picture takeaways that they could apply in the classroom. 


Class 2 Discussion Forum: Discussion 


Across all modules and prompts examined, much of class 2’s discussion was characterized by “back- 
patting” behavior: Students agreed with and praised the ideas of others without adding new thoughts to 
the conversation. This tendency toward “back-patting” is partially responsible for class 2’s low to 
medium Community Building (3.1) and Critical Thinking (2.9) scores. 
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Examples of “back-patting” can be found in module 2’s preferred prompt discussion: 


“| like that you thought about not just the language benefits, but how the pressures 
of each situation might affect a student's comfort level speaking. This is such an 
important part of making our classrooms feel safe, and is necessary for each student 
to grow his or her skills at the appropriate pace for him or her.” 


“| completely agree that [EL Student 1] would be successful in partner activities given 
she receives feedback well and enjoys working with her peers. Furthermore, the 
learning centers would really give her the time to seek help or suggestions on her 
work when needed. [EL Student 1] is a wonderful student :)” 


The instructor contributes only 4 replies to this discussion; while this is the most frequently she 
contributes to any discussion, her rate of response is still only 36%. It’s possible that her low 
participation may have contributed to this praise-focused pattern of discourse: Without examples of 
high-quality replies from an instructor, students may have lacked models of good discussion board 
conversation. In addition to being infrequent, the instructor’s responses here failed to model critical, 
productive discussion behavior. In this example, she engages in “back-patting”: 


“Your rationale for choosing Pick-a-Stick is very interesting. | agree that the fact that 
[EL Student 1] follows rules (and possibly feels comfort in and safe with the order 
that those rules provide) would make him more willing to participate when his name 
is chosen.” 


Class 2 received low Instructor Feedback scores across all preferred prompts (for an average of 2), and 
medium scores across the non-preferred prompts (for an average of 3). The teacher participated with 
fairly steady frequency (between 1 and 4 times) in each discussion forum, but the smaller number of 
student responses to non-preferred prompts increased her average rate of reply, giving her a greater 
influence on the direction of the related discussions. See Discussion Forum Participation Frequency 
Chart. 


This discussion question (describing a project, its potential problems, and a solution to one of the 
problems) emerges as a very successful prompt. 


Just as one prompt can raise the level of discourse, another prompt can lower it: Module 4’s preferred 
prompt asked students to describe a situation in which their affective filter came into play, and the 
students of class 2 responded to the personal, narrative quality of this prompt by sharing replies that 
were more oriented toward “back-patting” than other discussions. For example: 


“Thanks for sharing about your experience with foreign exchange students. | 
appreciate the illustration in how a supportive environment can be a powerful asset 
in learning a new language, doing well in school, and feeling comfortable in class.” 


“| like the idea your teacher implemented to sing with the class. That makes the 
environment more lighthearted, and lowers anxiety levels in students. | think it’s a 
great idea | would like to implement with my EL learners. :)” 
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Although the prompt did not promote critical thinking, it was very effective in encouraging students to 
reflect on their experience (scoring a 2 for Critical Thinking and a 5 for Reflection). 


Comparison of 2015 Discussion Forums to 2014 Discussion Forums: Discussion 


As mentioned in the Prompt Selection discussion, 2015’s introduction of multiple prompt options may 
have helped students feel more engaged in their studies by empowering them to make a choice in what 
they learn and how they learn it. In 2014, the lack of options seemed to make students take a more 
passive role in participating in the discussion forums. 


The 2015 curriculum includes discussion forum prompts in all 8 modules, starting from module 1. This 
makes group discussion a more consistent presence in the class than it was in 2014, when discussion 
forum prompts were only included in modules 2, 3, 4, and 5. Ongoing engagement in the discussion 
forum may give students a stronger experience of being part of a larger group, and may also help them 
feel more accustomed to and thus more comfortable with participating in the discussion. 


In 2014, a correlation was observed between low participation by the instructor and the class’s failure to 
engage in critical, consensus-building discussion. A similar correlation can be observed between the 
participation rate of class 2’s instructor and her class’s performance. Just as last year’s instructor posted 
an average of 1.7 times per forum, class 2’s instructor also posted minimally, between 1 and 4 times per 
forum. And in both cases, student posts often consisted of “back-patting,” or simple expressions of 
appreciation or acknowledgement of other students’ ideas, limiting the growth and the complexity of 
the conversation. On the other hand, class 1’s discussion board performance in 2015 suggests that an 
instructor may also be able to positively influence the course of a conversation by modeling critical and 
engaged posting behavior and by simply having an active presence on the forum; i.e., high participation 
by an instructor might make students feel that their own participation is being monitored and 
evaluated. 


Assignments 


The Sample. The selection of student work had two components. First, for each class, we randomly 
selected two participants from whom we would assess course assignments. Second, we selected 
assignments. When possible, we selected assignments similar to those assessed in 2014, but because 
the assignments were substantially different this year, we chose new assignments in order to reach a 
total of six assignments. See Table 9 in Appendix D. The following table shows the number of work 
products sampled by type and group. 


Assignment Sample Selection 


Class 1 Class 2 Total per 
Type 
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The Framework. The assignments and associated student work products were assessed on ten indicators 
across three criteria. Each indicator was scored ona scale of 1-5, with 5 as the highest score. The 
evaluations were derived from student work and instructional practice indicators in the Danielson 
Framework (Danielson 2013). See Appendix A. 


Class 1 Assignments: Findings 


e@ Class 1 had medium to high scores in Assignment Design (3.7) and Student Response (4.6), but low 
scores in Instructor Response (2). 

e Low scores on two related indicators brought down the overall Assignment Design and Instructor 
Response scores. Because there was no clear grading rubric, every assignment received the lowest 
possible score (1) for the following indicators: “Assignment rating criteria are specified” (which is 
factored into the Assignment Design score) and “Instructor ratings reflect the use of clear criteria 
and are applied fairly across a range of responses” (which is factored into each class’s Instructor 
Response score). If the Assignment Design score were recalculated without factoring in the “rating 
criteria” indicator, it would be substantially higher: 4.4 (compared to 3.7). Likewise, class 1’s 
Instructor Response score would be 2.5 (compared to 2) if it were recalculated without factoring in 
the “clear criteria” indicator. 

e Both students examined received a passing grade or “P” on all six assignments sampled. Students 
received the same grade for work of varied quality. 

e The quality of the instructor’s feedback varied widely; scores for the “Instructor provides actionable 
feedback” indicator ranged from 1 to 5. 


Class 1 Assignments: Discussion 


In some cases, variations in instructor feedback may be related to variations in types of assignments. For 
example, it may be easier for an instructor to give “actionable” feedback on an assignment focused on 
designing classroom experiences than on an assignment focused on describing a pair of students, like 
the module 2 WikiProject, Extending Student Profiles.” The instructor’s reply to Student 1 for her 
module 2 project is: 


“Great to learn more about Jorge and Ronan. This is going to be so helpful as 
we start designing our classrooms and short projects.” 


The instructor’s feedback to Student 1’s work on the module 4 WikiProject is much more actionable. 
Among other questions, the assignment asks students to discuss “two things you could do to get a 
better idea of whether or not the [CELDT] score is an accurate measure.” The instructor writes: 


“| appreciate the analysis of each score associated with students’ 
personalities, learning styles, and interests. With Vicente, | wonder how you 
might go about testing transfer and receptive vs. productive. Have you used 
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something like this in the past? If you haven't used something in the past, how 
might you go about testing these?” 


Given that the module 4 assignment called for students to plan and consider future actions, it may have 
been more natural for the instructor to provide actionable feedback. However, in other cases, the 
quality of instructor feedback does not appear to be related to assignment design. For the module 7 
WikiProject, Student 1 received feedback that was substantive and actionable: 


“Can you elaborate a little bit on what about this type of research is authentic. 
You're getting there, but just to make it clear exactly which elements you 
think and why would be helpful for me. Also, do you have any suggestions 
here for improvement in making it more authentic? Thank you for your work!” 


However, Student 2 received low-quality feedback for the same assignment: 


“Such great questions posed and suggestions for the projects you've 
reviewed. You did a great job with being kind, helpful, and specific in your 
approach. Thank you for taking the time.” 


Here, it seems that the instructor’s feedback may have been tailored to the quality of student work; 
student 2’s project was longer, more detailed, and more carefully thought out than student 1’s. 


Although all student work sampled was fairly high-quality (student 1’s average Student Response score 
was 4.2; student 2’s score was 4.8), there was some variation, but the pass/fail grading system did not 
acknowledge any variation. Student 2 consistently turned in assignments that were longer and more 
detailed than his peers’ work, but his grade did not reflect this; he earned the same “P” as student 1, 
who generally fulfilled assignment requirements but did not seem to put in as much time or effort. 


Just as a discussion forum prompt can raise or lower the level of student discourse, so too can an 
assignment. For example, although the module 7 WikiProject was generally well designed (it made a 
complex task simple by breaking it down into a list of questions), the content of some of these questions 
sometimes brought the student dialogue down to a “back-patting” level. In fact, the last question of the 
survey explicitly invites “back-patting”: “What is one final note of encouragement or genuine support 
you can offer to your colleagues?” Students follow directions, giving each other feedback that is 
enthusiastic but not critical or useful: “Can | be in your class? | would be so stoked to have a project like 
this to dive into!” 


The module 6 WikiProject, Instructional Strategy Plan, was a strong assignment because it required 
students to explain their thinking and guided them in doing so. It asked students to annotate a project 
plan they created in a previous module with notes indicating the instructional strategies they planned to 
use to support English Learners, and provided them with a clear list of what to add: 


1. Two instructional strategies that will help all EL students better 
understand and engage in the learning 

2. Two strategies you will use to check for their understanding 

3. Two strategies or supports designed with specific students in mind. 
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In order to fulfill the assignment, students had to apply their knowledge to a classroom setting and 
clearly explain their thinking. For example, student 2: 


“EL strategy: To help students create an excellent public education piece, | 
would ensure a thorough brainstorming session. First | would organize centers 
where groups could visit pods where information was set out describing 
previous successful / innovative / famous etc. public awareness raising events. 
Once their imaginations were piqued by this exercise, a group brainstorming 
session to dredge the collective group mind for ideas would be carried out 
with each group having a large piece of paper, and doing a table version of 
chalk talk — all writing down their ideas for a few minutes on the paper. From 
here, they could discuss and eliminate ideas they have doubts about. Sharing 
with the group and gathering critique and tips to further refine the results of 
the brainstorms would be helpful.” 


This project was not alone in using a list to clearly present instructions; many assignments used bulleted 
and numbered lists to give students a clear idea of what is expected of them and the steps they will 
need to take. 


Class 2 Assignments: Findings 


e@ Class 2 had medium to high scores in Assignment Design (3.7) and Student Response (4.3), but low 
scores in Instructor Response (1.9). These scores were very close to class 1’s scores. 

e@ Aswas the case with class 1, the “clear criteria” indicator lowered class 2’s instructor response 
score; if it were recalculated without factoring in this indicator, it would be 2.3 (rather than 1.9). 

e@ One of the students (student 3) examined received a passing grade or “P” on all six assignments 
sampled. The other student (student 4) received “P” grades on three assignments, while the grade 
boxes for her other three assignments were left blank. It is unclear what these blank spaces mean; 
in one of the instances, the instructor provided very positive feedback (“excellent job on identifying 
what the learning goals are for your students”) so it seems unlikely that the student failed the 
assignment. In the other two cases, the instructor did not leave any feedback on the assignment; it 
is possible she did not review the assignment. 

e Aswas the case with instructor feedback in class 1, the quality of the class 2’s instructor feedback 
varied widely; scores for the “Instructor provides actionable feedback” indicator ranged from 1 to 5. 


Class 2 Assignments: Discussion 


As with class 1, variations in instructor feedback may be related to the quality of student work and the 
nature of the assignment. Class 2, taught by two instructors, introduced another potential reason for 
variations. While class 1’s instructor replied to 100% of student projects sampled, the instructors of class 
2 failed to reply to 3 of the 12 projects sampled, giving them a reply rate of 75%. 


From the replies sampled, the average reply quality for each class 2 instructor was similar, and the 
instructors often took the same approaches to responding to student work. For example, when 
responding to module 4 WikiProject assignments, both Instructor A and Instructor B drew from their 
professional teaching experience: 


21 


Instructor A: “| thought that your analysis of Sofia's high score as being a 
result of her intrinsic motivation was interesting because | have seen that 
occur with students when | have administered the CELDT in the past. Some 
students see it as a reflection of who they are as students, and it is those 
students who really want to "do well" on the test (despite efforts to explain to 
them that the CELDT is only one of many gauges of language proficiency).” 


Instructor B: “Those inconsistencies you noticed with Djeyma happen quite 
often. The CELDT is given to all students whose parents indicated on their 
home language survey that a language other than English is spoken at home. 
Students are tests in the fall, results are available in the winter, and from 
there if the student scored high enough he/she can be redesignated. The EL 
coordinator must get a teacher recommendation, take a look at the student's 
grades from the previous year, as well test scores from the ELA. But, all this to 
say the process is long, complicated and can be flawed. Looking at alternative 
forms is essential to try to gain understanding and figure out how to support 
our student.” 


The constant and evolving presence of imaginary EL students throughout the curriculum seemed to help 
motivate students and help them practice applying ideas to a classroom setting. Students started with a 
Creating Our Class WikiProject in module 1, where they created a pair of students, one EL and one 
native speaker. In module 2, they created Extended Student Profiles, and wrote vivid descriptions and 
backgrounds for each student. And throughout the class, these students were available as a tool when 
students needed to practice applying concepts. 


In module 2’s Extended Student Profile, student 3 wrote about her imagined EL student, Ivette, with a 
high level of intimacy: 


“\vette understands the expectations of her when she can see them written 
down or they are communicated in one-on-one settings. She works best in 
small group settings, separated from her friends, and given extended time to 
work on assignments and turn in work.” 


This familiarity with the specific needs of an individual EL student supported students as they worked 
their way through the modules. In module 6, students were asked to describe the instructional 
strategies they will implement in their 3-day projects; part of the assignment is describing how certain 
strategies will work for individual students. Student 3’s familiarity with Ivette helps her give a clear, 
carefully reasoned answer: 


“A strategy that would work well for Ivette is to provide her with specific 
guidance for the annotated bibliography such as resources like bibme.com as 
well as an outline or model of what an annotated bibliography looks like so 
she would be able to reference the model for her work, perhaps in a fill-in- 
the-blank fashion.” 


The addition of these imagined students to the curriculum contributed to high scores across the 
modules for the “Assignments assess student capacity to implement course knowledge and/or strategies 
within their own classrooms” indicator (4.3 average). 
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Comparison of 2015 Assignments to 2014 Assignments: Findings 


e@ The 2015 Instructor Response scores (2 for class 1, 1.9 for class 2) were substantially lower than 
the corresponding 2014 score (3.6), while the other two scores were fairly similar. The 2015 
Assignment Design score (3.7) was very close to the 2014 score (3.5), and the 2015 Student 
Response score was higher (4.6 for class 1, 4.3 for class 2) than the 2014 Student Response score 
(3.9). 

e Asin 2014, course assignments did not provide a clear rating system for high-quality work. This 
lack of a grading rubric was again a weakness that lowered the overall average Assignment 
Design score. 

@ Onarelated note, the instructors graded on a pass/fail basis as they did in 2014. 


Comparison of 2015 Assignments to 2014 Assignments: Discussion 


As in 2014, 2015 course assignments failed to provide a clear rating system for high-quality work, and 
the binary pass/fail grading system made it difficult to ascertain the instructor’s use of clear criteria. 


There was a slight increase in the Assignment Design score (3.5 to 3.7), and one factor may have been 
higher scores on the “Assignments assess student capacity to implement course knowledge and/or 
strategies within their own classrooms” indicator. As mentioned previously, the inclusion of imaginary 
EL students throughout the curriculum made it easy to add an aspect of classroom implementation to 
any assignment. 


Like 2014’s curriculum, 2015’s curriculum included a final Presentation of Learning (POL), but the 
prompt was different. While the 2014 assignment presented a list of ten open-ended questions (for 
example, “Describe how you plan to establish an environment for learning in your classroom that 
supports multiple cultures and English learners”), the 2015 assignment was much more focused. 
Because of the way the 2015 assignments were structured around the imaginary students and the 
projects planned for them, the POL questions were able to assess a broad range of subject mastery by 
asking just five questions about the imagined students and their projects. 


2015 assignments included more partner and group work than 2014 assignments. Except for the final 
module’s POL, every 2015 WikiProject was a partner project, and the only individual assignments were 
Reflection writing prompts. While this gave 2015 students a great deal of experience working in teams, 
it may have made it more difficult for instructors to accurately assess individual student performance. 


Survey Results 
The Sample 


Based in California, High Tech High (HTH) is a public charter school organization. With its cluster of 
schools spanning kindergarten through Grade 12, HTH provides a rich clinical context for educator 
training and is approved by the state to certify its own teachers. In 2014 and 2015, HTH offered online 
classes designed to fulfill California’s requirement that pre-service preparation programs provide novice 
teachers with content to support English Language Learner students (ELs). At the end of the 2015 
summer term, a link to an online survey (see Appendix B) almost identical to the one used with the 2014 
summer class was uploaded to the HAIKU sites of the two 2015 classes. Students were asked to 
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complete the survey as their last task for the class, after they received instructor feedback on their final 
project. The response rates for the 24 students in the 2014 class and the 20 students in Class 2 (2015) 
were both 100%. Class 1 (2015) had a response rate of 89%; 16 of the 18 registered students responded. 
Given the substantial level of survey participation, we can be confident that the results reflect the 
opinions of all students taking the courses. 


The Findings® 
Novice Teachers’ Certification Areas and Teaching Experience (Tables 1a — 1c) 


e@ The 2015 classes had more than double the percentage of teachers at the elementary level than did 
the 2014 class, with Class 1 having 38% and Class 2 having 45%, compared to the 2014 class with 
only 17%. 

e About 30% of the teachers in both 2015 classes were certified in Special Education but only 17% of 
the 2014 teachers held that certification. 

e For novice teachers certified at the secondary level, Class 2 had a substantially higher percentage of 
Special Education teachers (55%) than did Class 1 or the 2014 class which both had 20%. 

e@ Class 2 had a much lower percentage of secondary level novice teachers certified in the Humanities 
(27%) compared with Class 1 (60%) and the 2014 class (50%). 

e Inthe 2014 class, none of the teachers had any teaching experience. Although to a large extent that 
was the case with the 2015 classes as well, a small group of the 2015 teachers taught in summer 
school in 2015. Only two students in Class 1 (13%) did so, far fewer than the 7 (35%) in Class 2. 


Novice Teachers’ Demographics (Tables 2a — 2e, 14 - 17) 


e Incontrast to the 2014 class that was split evenly between males and females, the majority of 
teachers were female in the 2015 classes, reaching 75% female in Class 2. That the teachers in the 
2014 program were 50% male was unusual since that figure is considerably higher than the national 
proportion of male teachers in K-12 settings. 

e Asin 2014, the majority of 2015 novice teachers were non-Hispanic white, but the percentage of 
Hispanic/Latino teachers was higher in the 2015 classes. Teachers of color made up about 45% of 
the 2015 classes compared to 38% for the 2014 class. 

e@ Class 2 is substantially younger® (average age = 28 years) than both Class 1 and the 2014 class where 
the average age was about 33 years in each. Class 2 is also more homogeneous with regard to age 
than Class 1 and the 2014 class. 

e The majority (68%) of the novice teachers in Class 2 are 30 and younger. In Class 1, half the teachers 
are in that age group. Only 38% of the 2014 teachers were 30 and younger. 


Interpreting the Course Evaluations 


Two issues should be considered when assessing the ratings for different courses. First, the mean of a 
scale being at or slightly above its midpoint does not reflect a satisfactory situation. Using the Course 


5 Tables related to the Survey results may be found in Appendix B. 

6 In order to evaluate the meaningfulness of the difference between two means, a conventional rule of thumb is to consider 
differences of about half a standard deviation in size as moderately important. Differences much smaller than the half a 
standard deviation benchmark are regarded as more trivial, while those considerably larger indicate a substantial effect. 
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Experience scales to illustrate, the midpoint (3) represents Undecided, so that a mean score of 3.4, for 
example, indicates sentiment somewhere between undecided and moderate agreement, a rather 
lukewarm endorsement of the experience. Mean ratings below 3 indicate a negative opinion. 


Secondly, the demographic profiles of the novice teachers in the three classes (1 in 2014 and 2 in 2015), 
as well as their areas of certification were sufficiently dissimilar that attempting to untangle the possible 
effects of course attributes and participant characteristics must be undertaken with caution. 


Finally, the fact that the teachers were rating their experiences through a course link raises the 
possibility that their responses may have been colored by their concern about confidentiality. 


Course Experience Ratings (Figure 1, Tables 3 — 5, 13 - 17) 


@ Onthe Instructor Engagement scale, the virtually identical mean ratings (4.6) of the two 2015 
classes were considerably higher than the 2014 class mean (4.1). 

e Similar results were observed in the case of the Culture of Learning scale, where the 2015 classes 
each had a mean of 4.0 while the 2014 mean was 3.5. 

e For the Course Design scale, the pattern of results was similar again. Here, too, the 2015 classes had 
similar means (about 4.4), substantially higher than the mean of the 2014 class (3.8). 

e Ofthe three Course Experience scales, the average rating across the three programs was highest for 
the Instructor Engagement scale. 


Effectiveness of Course Components (Tables 6 - 7b, 13 - 17) 


e For all goals (learning course content, learning how to teach course content to students, and 
enabling the implementation of knowledge and/or strategies directly into teachers’ classrooms), the 
components rated as most effective, on average across the three classes, were Readings, Videos, 
and Reflections/Module Reflections.’ 

e Discussion Boards also made the “most effective” list, but only for the 2015 classes. The 2014 class 
gave this component ratings that were a full standard deviation lower than those of the 2015 
teachers. 

e@ Although not as marked as the difference in Discussion Boards ratings, in general, the 2015 classes 
tended to rate the components more favorably than did the 2014 class. In addition to the Discussion 
Boards component, this was most apparent in the Individual Wiki Projects/Partner Team 
WikiProjects component and the Group Work/Partner Team Work components. The last is 
particularly noteworthy because the 2014 effectiveness ratings for Group Work averaged about 2.3 
across the three goals. Not only was this the lowest effectiveness rating of any component in 2014, 
it was well below the scale midpoint of 3. 

e There was a slight tendency for 2015 Class 1 to give higher ratings to some components than Class 2. 
This was most noticeable in the ratings for the Partner Team WikiProjects component and the 
Partner TeamWork component. In addition, Class 1 had a somewhat higher mean on the global 
Course Components Effectiveness scale (4.2) compared to Class 2 (3.9). Both were higher than the 
mean of the 2014 class (3.5). 


Preparation Beliefs (Tables 8 - 11, 13 - 17) 


7 As described earlier in Survey Components and Constructs, half of the components kept the same name in 2015 and half were 
renamed. 
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e@ HTH 2015 and 2014 classes rated their level of preparation similarly for many items. There were 
however some notable exceptions. One of the biggest differences observed was in the case of Item 
# 18 (Engage in planning and problem solving with colleagues) where the 2015 classes had 
substantially higher preparation scores than did the 2014 class. The 2015 classes also had 
meaningfully higher scores on Item # 8 (Relate classroom learning to the real world) and Item # 17 
(Evaluate and reflect upon your practice to improve instruction). The 2015 classes had somewhat 
higher preparation scores on Item # 13 (Choose teaching strategies to meet different student 
needs). 

e@ Forsome preparation items, Class 1 felt better prepared than did Class 2 and the 2014 class. Class 
1’s means for Item # 7 (Use instructional strategies that promote active student learning) and Item # 
9 (Understand how students’ social, emotional physical, and cognitive development influence 
learning) were substantially higher than the means of the other two classes. 

@ On Item #11 (Teach in ways that support English language learners) Class 1’s preparation ratings 
were somewhat higher than Class 2’s and substantially higher than those of the 2014 class. 

e With regard to Item # 11, 100% of the teachers in all three classes felt they were adequately 
prepared to teach in ways that support English language learners. 

e For Technology Use Preparation, the 2015 classes reported slightly better preparation than did the 
2014 class. 


Efficacy Beliefs (Tables 12, 13 - 17) 


e@ The Personal Teacher Efficacy (PTE) scores were similar for all three classes with an overall mean of 
3.9. This is almost a full scale point above the scale mean of 3, indicating that the teachers had a 
fairly positive feeling about their own teaching efficacy. 


e@ The General Teaching Efficacy (GTE) scores were also similar for all three classes with an overall 
mean of 2.5, indicating that the teachers had a fairly low level of concern about the effect of 
external factors on their teaching efficacy. 


Correlations: Course Experience, Course Components Effectiveness, Preparation, Teacher Efficacy, and 
Suggested Revisions (Table 13) 


e@ The more favorably teachers viewed their course experiences in terms of instructor engagement, 
culture of learning, and course design, the more positive their perceptions of course components 
effectiveness were. This relationship was strongest in the case of Culture of Learning where the 
correlation with Component Effectiveness was .74. 

e Assessments of course components effectiveness were strongly related to teachers’ perceptions of 
preparedness. The higher the effectiveness assessment, the more prepared the teachers felt. 

e@ Personal teacher efficacy was meaningfully related to the course experience of culture of learning, 
and also related, more strongly, to views of course components effectiveness and how prepared 
teachers felt. 

@ General teacher efficacy was not meaningfully related to any of the other variables. 


SubGroup Analyses 


This section reports the results of analyses comparing different demographic and experience groups. 
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Sex Comparisons (Table 14) 


e  =Ratings for males and females were similar except for Culture of Learning where the females had 
substantially higher ratings than males. 


Age Comparisons (Table 15) 
e@ There were no age differences observed. 
Ethnicity Comparisons (Table 16) 


e@ The teachers of color had more positive feelings about their teaching efficacy compared to the 
white teachers. 


Comparisons between Teachers with and without Summer School Teaching Experience 
(Table 17) 


e@ The teachers with summer school teaching experience gave higher ratings to Instructor 
Engagement and Culture of Learning than those without that experience. 

e@ The teachers with summer school experience had a lower PTE mean than those who did not teach 
in summer school. 


Discussion of Course Changes, Survey Findings, and Artifact Analyses 


Changing the Course and Engaging the Faculty 


In summer 2014, the partners launched Phase 1, an 8-week course that focused heavily on reading 
research and current publications on language acquisition theory designed to raise novice teachers’ 
awareness of the issues involved in teaching English Learners. Comments from participating New York 
and California novices (e.g., “I now teach EL students and do not have resources from the course to help 
me because | did not get a solid grasp of how to apply the strategies shown in the videos or laid out in 
the readings.”) raised important questions for program designers. Through the collection of these 
anecdotal data coupled with a rigorous comprehensive evaluation completed by a team based at New 
Visions (Trachtman, Koenigsberg, & Zheng, 2014), six key areas were identified in need of improvement. 
An online education consultant worked with one of the course instructors to design the following 
changes for implementation in Phase 2, the summer 2015 course. [Appendix Al provides a description 
of both the 2014 and 2015 key objectives and learning goals for each module; detailed guides to each 
module for both years may be found in Appendix A2.] 


1. From Groups to Partner Teams 
A. The Phase 1 course had encouraged group work in an effort to build a sense of community; 
however, by changing the groups each week and providing relatively little timely feedback to 
group participants, the novice teachers were not able to build their desired and desirable 
connections with peers. 
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B. Inresponse, in the Phase 2 course, instructors assigned 2-teacher partner teams to work 
together on a course-long project. Partner Teams remained together for the entire class and 
were encouraged to meet regularly by phone, Skype, or in person. In their required weekly 
reflections, novice teacher partners reported on their experiences, sharing confidentially with 
instructors their frustrations and achievements. When intervention was needed, the instructor 
worked with the specific Partner Team to facilitate a new communication process, helping the 
team to set clear assignment expectations and roles. 


2. From Weekly Assignments to a Single OverArching Project 

A. The Phase 1 course required weekly written assignments focused on the readings and research 
materials assigned that week. The strategy seemed to emphasize learning theory and the 
acquisition of new knowledge through the preparation of written summaries. 

B. Inthe revision, the Phase 2 course modeled a project-based learning approach with students 
assigned a series of tasks that built on each other, culminating in the development of a 3-Day 
unit or set of lessons that participants could use with their students in the fall. Beginning with 
their invention of two fictitious English Learners in the first week, novice participants began to 
develop an understanding of important issues related to students’ culture, families, and 
individual interests; they also learned about the types of tests that ELs would need to pass to 
demonstrate language proficiency. After learning specific instructional strategies, participants 
were asked to name and explain the specific strategies that would work best with their 
(fictitious) students. Finally, after spending time learning how to provide specific feedback, 
participant peers reviewed each other’s work and offered constructive and actionable next 
steps. 


3. From Discussion Questions to Real Discussions 

A. The Phase 1 course prompted participants with weekly discussion questions that related to 
research read and videos watched during the content portion of the week. In their postings they 
often summarized what they had read; peer responses to posts did not generally extend or 
deepen the conversation. Further, few discussions moved past the initial response posts. 

B. The Phase 2 course revised the discussion prompts to focus participants on their own personal 
experiences with the weekly topic. For example, rather than reflecting on Stephen Krashen’s 
Affective Filter Hypothesis, the prompt asked participants to share an uncomfortable moment 
when their own “affective filter” got in their way. The prompt was tweaked in the hope of 
inspiring lengthy and detail-rich posts that would encourage multi-participant dialogue. 
Instructors developed strategic “moves” for deepening discussions to facilitate the development 
of a dynamic learning community. 


4. From Abstract to Actionable 
A. The Phase 1 course focused on the theory of language acquisition leaving participants to infer 
how best to implement the theory in their own classrooms. They were invited to watch videos 
of other teachers teaching English Learners and evaluate the observed instruction. The video 
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watching strategy reflected the course’s implementation during the summer when the novices 
were not able to observe actual teaching at schools. 

The Phase 2 course required participants to dig past the theory by reviewing proven strategies 
and then deciding which strategies would work best for their specific students. By having 
novices “create” their own students at the beginning of the course, the project team believed 
the teachers would feel an important sense of responsibility and commitment to these students, 
and, vested in their success, would make curricular and instructional decisions based on their 
needs. While real students continued to remain out of sight, instructors had designed a process 
to mimic the approach used by experienced teachers to design effective instruction: begin by 
knowing your students well. 


5. From Anonymity to Collaborative Engagement 


A. 


The Phase 1 course provided few opportunities for structured contact between students and 
instructors other than the feedback exchanged on assignments and instructor posts on 
discussion boards; while some students reached out to their instructors via email, the online 
delivery of the course appeared to limit interaction and connectedness. 

The Phase 2 course featured the same instructors working hard to be perceived as real people 
attentive to participants and their learning. The instructors posted self-made videos welcoming 
the participants and orienting them to the week’s course of study. They posted pictures, brief 
bios, and hosted a Fun Things discussion thread to facilitate connections. The instructors were 
specifically trained in how to provide active feedback online, as they created an online presence 
that encouraged community. Through frequent postings that validated participants’ learning, 
instructors strove to deepen the discussions by pushing past the initial post, inviting participants 
to explore extended applications of the concepts emphasized that week. The project team made 
changes in how discussion questions were worded, and sought to increase interaction 
throughout the course. These changes were designed to transform the class culture from one 
oriented toward book knowledge to one that mirrored a collegial conversation about developing 
and implementing research-based instructional strategies for educating ELs. 


6. From Final Evaluation to Formative Data Collection 


A. 


The Phase 1 course asked instructors to collect weekly survey data but the data were not shared 
and discussed during the course. Instructors and the project team held 3 reflection 
conversations during the first summer, providing instructors with an opportunity to gain general 
insights into the students’ experiences. The final comprehensive evaluation, completed in 
December 2014, provided actionable feedback but results from this evaluation were not 
available until after the course had ended. 

The Phase 2 course used weekly reflections submitted directly to the instructors as a way to 
check on participants’ engagement with the online medium, instructors’ pedagogical choices, 
and the content of the curriculum. The project consultant actively supported instructors during 
the course to troubleshoot issues as they arose and/or take note of items that could be 
strengthened for the future. 
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Course Experiences. All three Course Experience scales -- Instructor Engagement, Culture of Learning, 
and Course Design -- were rated more favorably by the two 2015 classes than by the 2014 class. To a 
large extent, the more positive 2015 perceptions are undoubtedly linked to the extensive curriculum 
revisions made in response to student feedback and program issues identified in the 2014 evaluation. 
One major change involved the Discussion Board. In 2015, the novice teachers were given a choice of 
questions each week instead of only one, and new prompts were designed to focus participants on the 
relationship between their own experiences and the discussion topic. In addition, instructors developed 
strategies to deepen discussions and increase participation. These revisions met with success as 
evidenced by the artifact analysis of the Discussion Board. The improvement in the novice teachers’ 
perceptions of instructor engagement coupled with the higher artifact ratings of Discussion Board 
instructor feedback, as well as the more favorable student views of class culture paired with the 
improved artifact ratings of Discussion Board community building, lend validity to both sets of results. 
There was, however, a surprising finding in that the two 2015 classes had similar Instructor Engagement 
and Culture of Learning scores despite the fact that in the Discussion Board artifact ratings Class 1 had a 
higher Instructor Feedback score and a higher Community Building score compared to Class 2. Such 
incongruities sometimes occur in mixed methods research, and provide the researchers an opportunity 
to assess and perhaps better control some of the factors influencing the results. In this case, a possible 
explanation is that the self-report data collected via the survey captured participants’ perceptions of 
how they experienced the course in areas including, but not limited to, the Discussion Board. For 
example, the Instructor Engagement scale on the survey contained items related to instructor response 
to emails and feedback on assignments along with items that tapped student-instructor interactions 
such as those on the Discussion Board. An additional factor that may have contributed to the 
unexpected discordance between survey results and artifact ratings is the conflation of frequency and 
quality in the Instructor Feedback criterion used for the artifact analysis. [Restructuring the criterion’s 
indicators in the future may help paint a clearer picture of how students perceive those different facets 
of instructor feedback. ] 


Course Component Effectiveness. As in previous evaluations (Trachtman, Koenigsberg, & Zheng, 2014; 
Trachtman et al., 2014), the Course Experience scales were found to be related to teachers’ judgments 
of course component effectiveness. The generally more positive 2015 novice views of course 
component usefulness appear closely linked to the curriculum changes. 


Discussion Boards. Additional evidence of the enhancement of the quality of the 2015 
Discussion Boards discussed above was found in the analysis of the survey ratings of Course 
Components where the 2015 ratings for the Discussion Boards component were a full standard 
deviation higher than the 2014 ratings. This is quite an impressive improvement in teachers’ views of the 
effectiveness of the discussion boards in helping them to learn course content, learn how to teach 
course content to students, and enabling them to implement knowledge and/or strategies directly into 
their classrooms. 


Videos. The Videos component received some of the highest course component ratings from the 
novice teachers. There were a total of 22 new videos added in 2015 and 7 of the 8 modules received at 
least one of them. The Videos effectiveness ratings increased from 2014 to 2015, albeit modestly, and 
the improvement was most apparent for the goal of enabling teachers to implement knowledge and/or 
strategies into their classroom. This may be a reflection of the artifact analysis finding that the most 
successful videos were those that integrated aspects of classroom modeling with aspects of pedagogical 
theory. 
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Group Work/Partner Teams. In 2014, the novice teachers rated Group Work as the least 
effective course component across all goals (learning course content, learning how to teach course 
content to students, and enabling the implementation of knowledge and/or strategies directly into 
teachers’ classrooms). Its average rating of 2.3 was well below the scale midpoint of 3. A typical student 
comment in 2014 was: 


| found group assignments weren't very collaborative. "Make a presentation 
together" or "write a paragraph together" are activities that could be done 
individually. In most cases, one person wrote a part and slapped it together with 
a part someone else had wrote (this communicated via email drags out work an 
individual could do faster and more seamlessly). 


The curriculum redesign called for 2-teacher partner teams that stayed together for the entire course. 
They were assigned a term-long project that actually required both members of the team to participate 
together. The average rating for the Partner Teams component in 2015 was strikingly higher compared 
to the Group Work component in 2014. There were not many student comments in 2015 but one Class 1 
student did write the following, which was a far cry from the general sentiment in 2014: 


Although | dreaded an online course, | actually felt that | learned a lot. The 
content blocks were extremely helpful, and | enjoyed working with a partner. 


Class 1 had slightly more favorable perceptions of the Partner Teams component compared to Class 2. 
Since the assignments were the same, perhaps the teams tended to be more compatible in Class 1. 
More research is needed to determine optimal guidelines for team formation in online courses. 


Wiki Projects. The introduction of the term-long project may also have been responsible 
for the substantial increase in the 2015 novice teachers’ ratings of the Partner Team 
WikiProjects over their counterparts’ 2014 ratings of the Individual Wiki Projects. Analysis of the 
Assignment artifact suggested that the presence of imaginary EL students throughout the 
curriculum helped motivate the novices and helped them practice applying concepts to a 
classroom setting. The familiarity with the specific needs of their individual EL students provided 
support to the novices as they worked on Wiki assignments and probably contributed to their 
recognition of the WikiProjects effectiveness. 


Prepared to Teach. Further evidence that the change from group work to partner teams was 
successful came from the preparation beliefs held by the 2014 and 2015 classes. Although the 
Phase 1 and Phase 2 classes rated their level of preparation similarly in many areas, the most 
notable exception was the large difference between 2014 and 2015 responses to “Engage in 
planning and problem solving with colleagues.” The higher rating from the 2015 class apparently 
reflected their more positive experience with the Partner Team component compared to the 
2014 class’s Group Work. Other preparation items on which the 2015 class reported feeling 
considerably more prepared also seemed related to the Phase 2 course revisions. For example, 
participants’ responses to “Evaluate and reflect upon your practice to improve instruction” 
indicated more positive feelings of preparation in 2015 which may have been related to the 
novice teachers’ experience with the revised reflection topics and the more conversational tone 
in which they were asked to think about them. Similarly, the more favorable 2015 responses to 
“Choose teaching strategies to meet different student needs” were likely enhanced by the 
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newly-designed course project mentioned above in which partner teams invented fictitious 
English learners in the first week, then developed an understanding of important issues related 
to students’ culture, and, after learning specific instructional strategies, identified and explained 
the specific strategies that would work best with their (invented) students. 


For Technology Use, the 2015 classes reported slightly better preparation than did the 2014 class. 
However, future research should pay attention to discovering the relationships between pre-service 
teachers’ technological knowledge and their capacity to integrate technological, pedagogical and 
content knowledge, now called TPACK (Abbitt, 2011), when designing and implementing effective 
lessons and units. Further, it is likely that an additional relationship exists between teachers’ “TPACK” 
and their self-efficacy beliefs. 


Teacher Efficacy. Despite the important differences between the 2014 and 2015 classes in some 
preparation areas, all three classes had similar high Personal Teacher Efficacy (PTE) scores. In our VPSS 
research (Trachtman, Koenigsberg, Zheng, Cho, & Marlow, 2014), we noted the absence of an “online 
course effect” related to PTE scores. That is, even when teachers routinely gave low ratings to almost all 
aspects of particular courses, this “reverse halo effect” did not extend to the rating of PTE. Given that 
personal efficacy beliefs are probably strongly related to individual teacher characteristics and teaching 
experience, and therefore not greatly influenced by attitudes toward the course taken, this finding 
makes sense. 


There were two instances where we observed differences in PTE scores. The group of 2015 novice 
teachers who were teaching in summer school at the same time they were taking the EL course had a 
lower PTE mean score than did their counterparts who were not doing any teaching. This is not an 
uncommon phenomenon. When first faced with the challenge of actually teaching real students, not just 
learning about how to teach them, the confidence new teachers have in their ability can be seriously 
shaken (Veenman, 1984). A second difference in PTE scores was observed between ethnic/racial groups 
where teachers of color had more positive beliefs about their teaching efficacy than did white teachers. 
Although we have seen this effect in some prior evaluations (Trachtman & Koenigsberg, 2003-2008), it 
has not been observed consistently. More research is needed to see exactly which racial/ethnic groups 
tend to feel this way. Aggregating a number of different groups into the “of color” category results in 
the composition of this category being different in different studies. 


Class Size. Earlier, we reported on the 2015 novice teachers’ more favorable perceptions of instructor 
engagement and proposed that changes to the discussion forums were likely responsible. However, 
another factor may also have played a part. The class sizes in 2015 (16 students in Class 1 and 20 
students in Class 2) are smaller than the 24 students in 2014. Although there is considerable 
disagreement in the literature regarding what, exactly, is an optimum online class size (Orellana, 2006), 
the general wisdom suggests that larger classes increase instructor load leading to decreased instructor- 
student interaction. Given the strong relationship reported between the level of an online instructor’s 
interaction with students and students’ perceived learning, satisfaction, and sense of community (Baran, 
Correia, & Thompson, 2013), the smaller class sizes may very well have influenced the more favorable 
2015 ratings. Support for this possibility comes from the fact that the same educator served as the 2014 
class instructor as well as Class 1’s instructor in 2015 where the Instructor Engagement scores were 
substantially higher. It is impossible to disentangle all the variables involved in this comparison including 
the revised curriculum and the online experience and content familiarity gained by the instructor in 
2014, but the class size change needs to be considered. 


32 


HTH recognized the importance of class size and limited the two 2015 classes to 20 students each. Using 
a different analytic lens, the class size issue warrants further investigation because while smaller classes 
may be more desirable, along with economic constraints there are other reasons to not drop below a 
minimum class size. Particularly in an online class such as the EL course currently under review, it is 
important to have a sufficient number of students to interact effectively with one another. The course 
offered an excellent example of how too few students could detract from meaningful interaction. In 
reviewing Class 1’s discussion forums we found that when only a small group of novices chose to 
participate in a particular forum, they did not receive substantial critical feedback from the handful of 
peers in that discussion, and, because of the small number of students participating, there was minimal 
peer-to-peer interaction resulting in weak community building. 


Class Composition. In forming the two 2015 classes, HTH paid attention not only to class size but also 
took the novice teachers’ credential routes into account. An equal number of single subject teachers, 
multiple subjects teachers, and special educators was assigned to each class. However, other teacher 
characteristics were not considered. As it turns out the classes differed considerably in age with Class 2 
being a younger group. Class 2 was also more female. Despite these differences, the same syllabus was 
used for both classes and we did not see any evidence of attempts to differentiate instruction. 


Concurrent Summer Activity. Unlike the novice teachers in the 2014 class who had no contact with 
students while taking the course, in the 2015 class 9 of the novices were teaching in summer school. The 
majority of them (7) were in Class 2. Knowing how challenging teaching is for a novice, we wonder how 
the concurrent teaching experience impacted their module participation. As mentioned earlier, review 
of the discussion forums indicated that Class 2 received lower ratings on community building. Their 
posts were also rated lower than Class 1’s on critical thinking. Possibly, those ratings were influenced by 
the 35% of Class 2 who were teaching while taking the EL course. 


In addition to those who were teaching in summer school, all of the EL novice teachers were required to 
participate in Odyssey, a summer in-service program for HTH interns that started while the EL course 
was still running. In a very positive review of the EL course, one student commented: 


My only suggestion might be to begin a week earlier so that it doesn't overlap 
with Odyssey, if possible. 


And another student suggested: 


| would like to see more conscious integration between the Online EL class and 
the exercises of Odyssey. Seems like there could be a relationship productively 
between the two. 


Concluding Comments 


The 2014 and 2015 evaluations of the EL online learning modules reflect the commitment of the 
Schusterman Foundation to support practitioner learning and continuous improvement. Few course 
designers are given financial resources to iterate on their initial designs; fewer still benefit from a multi- 
year engagement by evaluators committed to studying the effects of curricular and pedagogical 
revisions. And so we conclude this report where we began, by celebrating the support of the 
Foundation, the two partnering organizations, High Tech High and New Visions for Public Schools, that 
incubated these ideas, the skill and commitment of course designers and instructors, and the 
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engagement of course participants who shared their experiences and gave us access to their insider 
knowledge. 


We learned much during this second evaluation, including how online learning outcomes increase by 
making the learning experience more personal, by giving participants more choice and voice in their 
assignments, and by connecting learners to their peers with assignments that demand collaboration and 
joint work. We also came to understand that the pedagogical choices made by instructors using the 
same curriculum are consequential, and that their pattern of presence and engagement catalyzes their 
students’ participation even in asynchronous online settings. 


Our evaluations also raise several questions related to online class size, group formation within the 
course, the effect on novice teachers’ learning when concurrently engaged with P-12 students, and the 
characteristics of the participating learners themselves. We look forward to continuing our inquiry into 
this field of study since we predict that it will grow exponentially as children, youth, and adults turn 
more frequently to technology to support their own learning. 


A final thought: we would like to return to Malcolm Knowles’ 5 adult learning theory assumptions and 
include a useful examination provided by ISpring® 


1. Assumption #1 (Self-Concept) 
Create learning experiences that offer minimum instruction and maximum autonomy. 
A major aspect of designing adult eLearning courses is having an eLearning support system to offer 
guidance and help, while still giving the eLearning tools and resources they need to learn on their 
own terms. Adult learners acquire new information and build upon existing knowledge much more 
effectively if they are encouraged to explore a topic on their own. While younger learners might 
need to be guided through the learning process, mature learners will typically get more out of the 
experience if they are able to work autonomously. This might come in the form of self-study or 
group collaboration projects that involve minimal instructor intervention. ELearning professionals 
can also offer simulations, scenarios, or games without prefacing them with any information. As 
such, the adult learners will have to explore the activity on their own, and decide which benefits and 
information they can take away from the eLearning experience. With that being said, you'll also 
want to have an eLearning support system in place if they need to ask questions or to overcome any 
obstacles that may be hindering the eLearning process. 

2. Assumption #2 (Adult Learner Experience) 
Include a wide range of instructional design models and theories to appeal to varied experience 
levels and backgrounds. 
Adult learners are more mature. Therefore, they have had more time to cultivate life experience and 
typically have a wider knowledge base. That means that you'll have to take into account that your 
adult learning audience is going to be more diverse, especially in terms of backgrounds, experience 
levels, and skill sets. While one adult learner may be well versed on how to search for resources 
online, another may have very little experience using the Internet. All of this must be considered 
when designing and developing your eLearning courses and eLearning activities. To appeal to 
different adult learners, it's often best to include a variety of different instructional design models 
and theories into your eLearning course or module. Survey your audience beforehand to determine 
any technical knowledge limitations they may have, as well as to assess their education levels. By 


8 https://www.ispringsolutions.com/blog/what-does-malcolm-knowles-know-about-adult-learning-theory/ 
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doing this, you will also be able to create eLearning experiences that are informative and engaging, 
rather than too challenging or boring. For instance, if your target audience includes a number of 
adult learners who may already know how to use multimedia, then including them in your eLearning 
course will boost its effectiveness and make it more immersive. 

Assumption #3 (Readiness to Learn) 

Utilize social media and online collaboration tools to tie learning to social development. 

As we get older, we tend to gravitate more toward learning experiences that offer some sort of 
social development benefit. For example, we are often more ready to challenge ourselves with new 
learning opportunities if we know it will help us to fine tune skills that pertain to our social roles. 
From an eLearning professional point of view, social media and online collaboration tools can help 
you to incorporate this assumption into your deliverables. Create activities that encourage adult 
learners to use sites like LinkedIn and Google Plus as invaluable tools. This can help them to not only 
build their social network, but also collaborate with those who share the same interests. 
Assumption #4 (Orientation to Learning) 

Emphasize how the subject matter is going to solve problems that an adult learner regularly 
encounters. 

Adult learners, essentially, need to know the why and when before they actively engage in the 
eLearning process. For example, they will not only want to know why they need to acquire specific 
information, but whether or not that information can be applied in the immediate future. Younger 
learners accept the fact that the knowledge they're acquiring today may not be used for quite some 
time. However, mature learners prefer to engage in eLearning experiences that help them to solve 
problems they encounter on a regular basis (in the here-and-now, rather than the future). So, you'll 
want to emphasize how the subject matter is going to help them solve problems immediately by 
offering real world examples and scenarios. 

Assumption #5 (Motivation to Learn) 

There must be a valid reason behind every eLearning course, module or educational activity. 
Motivation is key with adult learners. As such, you will need to motivate them to learn by offering 
them a reason for every eLearning activity, assessment, or eLearning module they'll need to 
complete. eLearning professionals must explain why a particular eLearning course is being taught 
and why an adult learner must participate in an eLearning activity, in order for the overall eLearning 
experience to be meaningful and engaging. For example, if you are asking adult learners to complete 
a group collaboration task, you should also clearly define that this exercise will help them to build 
their team working and communication skills, even after the eLearning course is over. While 
younger learners won't need to necessarily know the reason why they are required to participate in 
an activity, adult learners need to feel as though they are more involved in the process of learning. 
Otherwise, they will question the validity of the eLearning course, given that they don't see any real 
need for acquiring the new knowledge or skills. 
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Participant Survey 
Part |. Your Course Taking Experiences 
1. Course Experiences 


Please indicate how strongly you agree or disagree with each of the following statements for the ELL MODULES you 
completed online. 


Strongly Moderately Undecided Moderately Strongly 
Disagree Disagree Agree Agree 
1 2 3 4 5 
a. | felt comfortable using new technology (i.e., navigating HAIKU, browsing the Web, etc.). 
b. The Course Overview and Module Overview posted on HAIKU helped me understand the course requirements and grading 
standards. 


The HAIKU web site used in this course was well organized. 

In general, | found that using computer technology to learn is engaging. 

| enjoyed interacting with my classmates using HAIKU Discussion Boards. 

| felt comfortable sharing my opinions and ideas with my classmates. 

| felt comfortable sharing my opinions and ideas with my instructor(s). 

This course has made me sensitive to the social skills needed for online learning. 

The amount of work in this course was comparable to face-to-face courses on similar content. 
Overall, | was satisfied with my classmates’ level of participation in this class. 

The expectations of me as a student were clearly stated and | fully understood my responsibilities in the course. 
| would recommend this course to other students. 

| was Satisfied with accessing and navigating this course website. 

| was satisfied with the technology support response time and availability. 

My instructor(s) encouraged me to ask questions and provide feedback. 

While doing activities in this course, | felt like | was given choices and options. 

| didn’t feel very good about the way my instructor(s) interacted with me. 

| felt that my instructor(s) was/were available to me online when | needed her/them. 
| enjoyed the activities where | interacted with my classmates. 

My instructor(s) provided timely responses to my emails. 

My instructor(s) conveyed confidence in my ability to do well in the course. 

My instructor(s) provided quality feedback on written assignments. 

| was satisfied with my performance in this class. 


exeryvraposgo- rT se spao 


2. Course Component Effectiveness 


Please rate the effectiveness of each course component on a scale of 1 (least effective) to 5 (most effective). 


Course Component HELPED ME HELPED ME LEARN HOW TO | ENABLED ME TO IMPLEMENT 
LEARN COURSE TEACH COURSE CONTENT KNOWLEDGE AND/OR STRATEGIES 
CONTENT TO MY STUDENTS DIRECTLY INTO MY CLASSROOM 

Readings 

Videos 


Partner Team 
WikiProjects 


Partner Team 


Prior Knowledge Thinking 
Prompts 


Module Reflections 
Discussion Boards 
Presentation of Learning 


Part Il. Your Preparation to Teach English Language Learners 
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The next set of questions asks you to think about how prepared you felt to teach English Language Learners after completing 
the course. 


How well prepared did you feel you were to do the following? 


Very poorly prepared Poorly prepared Adequately prepared Well prepared Extremely well prepared 
1 2 3 4 5 

1. Teach subject matter concepts, knowledge, and skill in ways that enable students to learn. 

2. Understand how different students in your classroom are learning. 

3. Set challenging and appropriate expectations of learning and performance for students. 

4. Help all students achieve high academic standards. 

5. Planinstruction that builds on students' experiences, interests, and abilities. 

6. Evaluate curriculum materials for their usefulness and appropriateness for students. 

7. Use instructional strategies that promote active student learning. 

8. Relate classroom learning to the real world. 

9. Understand how students’ social, emotional, physical, and cognitive development influence learning. 


10. Identify students’ special learning needs and/or difficulties. 

11. Teach in ways that support English language Learners. 

12. Choose teaching strategies for different instructional purposes. 

13. Choose teaching strategies to meet different student needs. 

14. Help students become self-motivated and self-directed. 

15. Help students learn to think critically and solve problems. 

16. Use multiple forms of assessment to determine students’ strengths and needs 

17. Evaluate and reflect upon your practice to improve instruction. 

18. Engage in planning and problem solving with colleagues. 

19. Use media and other technology to make the subject matter more accessible and meaningful to students. 


Part Ill. Your beliefs about learning and teaching 


Please indicate how strongly you agree or disagree with each statement below. 


Strongly Moderately Undecided Moderately Strongly 
Disagree Disagree Agree Agree 
1 2 3 4 5 

1. When | really try, | can get through to most difficult students. 

2. If a student did not remember information | gave in a previous lesson, | would know how to increase his/her retention 
in the next lesson. 

3. The hours in my class have little influence on students compared to the influence of their home environment. 

4. The amount that a student can learn is primarily related to family background. 

5. The influences of a student’s home experiences can be overcome by good teaching. 

6. When a student does better than usual, many times it is because | exerted a little extra effort. 

7. If a student in my class becomes disruptive and noisy, | feel assured that | know some techniques to redirect him 
quickly. 

8. If one of my students could not do a class assignment, | would be able to accurately assess whether the assignment 
was at the correct level of difficulty. 

9. If a student masters a new math concept quickly, this might be because | knew the necessary steps in teaching that 
concept. 

10. When the grades of my students improve it is usually because | found more effective teaching approaches. 

11. When a student is having difficulty with an assignment, | am usually able to adjust it to his/her level. 

12. When a student gets a better grade than he/she usually gets, it is usually because | found better ways of teaching that 

13. A teacher is very limited in what he/she can achieve because a student’s home environment is a large influence on 
his/her achievement. 

14. If students are not disciplined at home, they aren’t likely to accept any discipline. 

15. If parents would do more with their children, | could do more. 


16. Even a teacher with good teaching abilities may not reach many students. 
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Part IV. Information about You 


1. Whatis your California Teacher Certification Area? 
(please check all that apply) 


Multiple Subjects (K-8) 
English 

History/Social Science 
Science 

Math 

Spanish 

Art 

Music 

Special Education 


2. Were you teaching in summer school this summer? 


3. | Whatis your age? 


4. Whatis your sex? 
@ Male 
@ Female 


5. What is your race/ethnicity? (please check the one option that best describes you) 
e@ = American Indian or Alaska Native 

Hawaiian or Other Pacific Islander 

Asian or Asian American 

Black or African American 

Hispanic or Latino 

Non-Hispanic White 

Biracial 


Part V. Additional Comments: Please share with us final thoughts about your experiences this summer with the ELL modules. 
Thank you for taking the time to participate in this survey. 
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Appendix A 
Rubrics for Analysis of Course Artifacts (Videos, Discussion Boards, Assignments) 


Videos 


The Framework. The evaluation rubric for the instructional videos was derived from the 2013 Danielson framework and 
educational research on the effective use of instructional videos in the classroom (Canning-Wilson & Wallace 2000; Cruse 2011; 
Danielson 2013). After determining the relevant Danielson domains, we established 3 broad assessment criteria, constructed 
multiple indicators, and assessed each on a scale of 1-5, with 5 as the highest score. We then determined a total criterion score 
by averaging all individual indicator scores. 


1. Designing coherent instruction 
a. Video is designed to engage participants and advance them through the lesson 
b. Video is appropriate to the learning needs of novice teachers 
c. Video is aligned to the learning objectives of the lesson 


2. Engaging viewers in understanding content 
a. Video is short, thought-provoking, and well-focused 
b. Video contains short chunks of information that can be easily absorbed 
c. Video demonstrates the how-to and how-not-to of the lesson* 


3. Video production and viewing 
a. Video shows pictures or visuals directly related to the information or lesson being presented 
b. Video has clear visuals and text that is easy to read 
c. Video is well produced with good quality sound, imaging and acting 


*Based on last year’s evaluation, the team revised the criteria for Engaging viewers in understanding content, collapsing 
‘Video instructor articulates the rationale of instructional choices” and “Video instructor reflects on their instructional 
choices” into a single indicator, “Video demonstrates the how-to and how-not-to of the lesson.” 


Discussion Boards 


The Framework. We utilized the 2014 discussion forum evaluation rubric again. As described previously (Trachtman et al, 2014), 
the rubric was modified from criteria initially created by TeacherStream (TeacherStream 2009). Each forum was assessed on 
eight indicators across five criteria and rated on a scale of 1-5, with 5 representing the highest score. The average score for 
each criterion was calculated afterwards. 


We also derived a student participation percentage based on the number of discussion threads started by participants 
compared to the total number of participants enrolled in the thread. A 100% participation percentage meant each participant 
created at least one new discussion thread in the forum. The participation frequency of the course instructor was determined 
from the number of instructor replies to participants’ discussion threads. 


1. Instructional Feedback 
a. When students post questions, the instructor responds quickly and directly to the student’s concerns 
b. Instructor contributes by posting opinions, feedback, and insight beyond what is offered in the module 
presentations 


2. Demonstration of Knowledge of Key Concepts 
a. Students’ discussion board postings relate to concepts from the module’s learning objectives 
b. Student posts add to the class’s understanding of the topic 


3. Community Building 
a. Peer to peer interaction is frequent, with postings that lead to ongoing conversations 
b. There is evidence of consensus building: students work together to create a product, construct new knowledge, 
and/or come to an agreement on some topic 


4. Reflection 
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a. Students share a synthesis of the learning experience, or describe how an aspect of the module’s content has 
personal value to them 


5. Critical Thinking 
a. Students pose questions, think through problems, and/or create connections between other students’ postings 


Assignments 


The Framework. The assignments and associated student work products were assessed on ten indicators across three criteria. 
Each indicator was scored on a scale of 1-5, with 5 as the highest score. The evaluations were derived from student work and 
instructional practice indicators in the Danielson Framework (Danielson 2013). 


1. Assignment Design 
Assignments match learning expectations, and expectations are clearly defined for each assignment 
Assignments include learning tasks that require high-level student thinking and invite students to explain 


a. 
b. 


their thinking 


Assignments assess whether students have learned the corresponding course content 
Assignments assess student capacity to implement course knowledge and/or strategies within their own 


classrooms 
Assignment rating criteria are specified 


2. Student Response to Assignment 
Student work reflects their knowledge of key concepts 
Student work clearly demonstrates how they arrived at conclusions via the use of evidence, reasoning, 


a. 
b. 


and/or problem solving 


3. Instructor Response to Student Work 


a. 


Instructor provides actionable feedback 


Instructor ratings reflect the use of clear criteria and are applied fairly across a range of responses 
Instructor feedback is substantive and identifies areas of student strength and areas of potential 


improvement 


Survey Results 


Table 1a Novice Teachers’ Certification Area by Class 


Certification Multiple Subjects Count 
(K-8) % within 
Class 
English Count 
% within 
Class 
History/Social Count 
Science % within 
Class 
Science Count 
% within 
Class 
Math Count 
% within 
Class 
Spanish Count 
% within 


Class 


2015 Class 


Class 
2015 Class 


2014 Class 


4 
16.7% 


Total 


Total 


Art 


Music 


Special Ed 


Multiple Subjects 
(K-8), Special Ed 


English, 


History/Social 


Science 


Science, Math 


Table 1b Secondary Level Novice Teachers’ Certification Area (3 Disciplines) by Class 


Certification Area 


Total 


Math/Sci 
Humanities 


Special Ed 


Table 1c Novice Teachers’ Summer School Teaching Experience 


Taught in Summer School 


Total 


No 


Table 2a Novice Teachers’ Sex by Class 


Sex Male 


Female 


Total 


Count 
% within Class 
Count 
% within Class 
Count 


Count 3 1 4 
% within 18.8% 5.0% 16.7% 
Class 
Count 0 0 1 
% within 0.0% 0.0% 4.2% 
Class 
Count 2 6 4 
% within 12.5% 30.0% 16.7% 
Class 
Count 3 0 0 
% within 18.8% 0.0% 0.0% 
Class 
Count 0 2 1 
% within 0.0% 10.0% 4.2% 
Class 
Count 0 0 1 
% within 0.0% 0.0% 4.2% 
Class 
Count 16 20 24 
% within 100.0% 100.0% 100.0% 
Class 
Class 
2015 Class 1 2015 Class 2 2014 Class 
Count 2 2 6 
% within Class 20.0% 18.2% 30.0% 
Count 6 3 10 
% within Class 60.0% 27.3% 50.0% 
Count 2 6 4 
% within Class 20.0% 54.5% 20.0% 
Count 10 11 20 
% within Class 100.0% 100.0% 100.0% 
Class 
2015 Class 1 2015 Class 2 
Yes Count 2 7 
% within Class 12.5% 35.0% 
Count 14 13 
% within Class 87.5% 65.0% 
Count 16 20 
% within Class 100.0% 100.0% 
Class 
2015 Class 1 2015 Class 2 2014 Class 
6 5 12 
37.5% 25.0% 50.0% 
10 15 12 
62.5% 75.0% 50.0% 
16 20 24 
100.0% 100.0% 100.0% 


% within Class 
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60 
100.0% 


Total 


10 
24.4% 
19 
46.3% 
12 
29.3% 
41 
100.0% 


Total 


Total 


23 
38.3% 
37 
61.7% 
60 
100.0% 


Table 2b Novice Teachers’ Race/Ethnicity by Class 


Class 
2015 Class 1 2015 Class 2 
Ethnicity American Indian or Alaska Count 0 0 
Native % within Class 0.0% 0.0% 
Hawaiian or Other Pacific Count 1 1 
Islander % within Class 6.2% 5.0% 
Asian or Asian American Count 1 0 
% within Class 6.2% 0.0% 
Black or African American Count 1 2 
% within Class 6.2% 10.0% 
Hispanic or Latino Count 3 5 
% within Class 18.8% 25.0% 
Non-Hispanic White Count 9 11 
% within Class 56.2% 55.0% 
Biracial Count 1 1 
% within Class 6.2% 5.0% 
Total Count 16 20 
% within Class 100.0% 100.0% 
Table 2c Novice Teachers’ Race/Ethnicity (2 Groups) by Class 
Class 
2015 Class 1 2015 Class 2 
Ethnicity 2 white Count 9 11 
% within Class 56.2% 55.0% 
of color Count 7 9 
% within Class 43.8% 45.0% 
Total Count 16 20 
% within Class 100.0% 100.0% 
Table 2d Novice Teachers’ Age by Class 
Age 
Class 
2015 Class 1 2015 Class 2 2014 Class Total 
Mean 32.50 27.95 33.25 31.34 
Median 30.50 26.00 32.00 30.00 
Std. Deviation 7.694 5.049 8.538 7.601 
Minimum 21 22 22 21 
Maximum 49 40 56 56 
Table 2e Novice Teachers’ Age (2 Groups) by Class 
Class 
2015 Class 1 2015 Class 2 
age 30 and under Count 8 13 
% within Class 50.0% 68.4% 
Over 30 Count 8 6 
% within Class 50.0% 31.6% 
Total Count 16 19 
% within Class 100.0% 100.0% 


Figure 1 Course Experience Scales 


Three scales were constructed on the basis of a factor analysis of the Course Experience items. 


2014 Class 
1 

4.2% 

0 

0.0% 

0 

0.0% 

1 

4.2% 

2 

8.3% 
15 
62.5% 
5 
20.8% 
24 
100.0% 


2014 Class 
15 
62.5% 
9 
37.5% 
24 
100.0% 


2014 Class 
9 
37.5% 
15 
62.5% 
24 
100.0% 


45 


Total 


1 
1.7% 
2 
3.3% 
1 
1.7% 
4 
6.7% 
10 
16.7% 
35 
58.3% 
7 
11.7% 
60 
100.0% 


Total 


35 
58.3% 
25 
41.7% 
60 
100.0% 


Total 


30 
50.8% 
29 
49.2% 
59 
100.0% 


46 


Instructor Engagement (alpha reliability = .90) 

My instructor(s) provided timely responses to my emails. 

My instructor(s) conveyed confidence in my ability to do well in the course. 

| felt that my instructor(s) was/were available to me online when | needed her/them. 
My instructor(s) encouraged me to ask questions and provide feedback. 

My instructor(s) provided quality feedback on written assignments. 

While doing activities in this course, | felt like | was given choices and options. 

| felt comfortable sharing my opinions and ideas with my instructor(s). 

| was satisfied with my performance in this class. 


Culture of Learning (alpha reliability = .77) 
| enjoyed the activities where | interacted with my classmates. 


| enjoyed interacting with my classmates using HAIKU Discussion Boards. 

Overall, | was satisfied with my classmates' level of participation in this class. 

| would recommend this course to other students. 

This course has made me sensitive to the social skills needed for online learning. 

The amount of work in this course was comparable to face-to-face courses on similar content. 
In general, | found that using computer technology to learn is engaging. 

| felt comfortable sharing my opinions and ideas with my classmates. 


Course Design (alpha reliability = .78) 

| felt comfortable using new technology (i.e., navigating HAIKU, browsing the Web, etc. 

The HAIKU web site used in this course was well organized. 

| was Satisfied with accessing and navigating this course website. 

The Course Overview and Module Overview posted on HAIKU helped me understand the course requirements and grading 
standards. 

The expectations of me as a student were clearly stated and | fully understood my responsibilities in the course. 

| was satisfied with the technology support response time and availability. 


Table 3 


Means and Standard Deviations of Instructor Engagement Scale for Different Classes 


(Scale 1- 5: Strongly Disagree to Strongly Agree) 


N Mean Std. Deviation 
2015 Class 1 16 4.6250 52042 
2015 Class 2 20 4.6063 .41611 
2014 Class 24 4.0677 .65211 
Total 60 4.3958 .60286 


Table 4 
Means and Standard Deviations of Culture of Learning Scale for Different Classes 


(Scale 1- 5: Strongly Disagree to Strongly Agree) 


N Mean Std. Deviation 
2015 Class 1 16 3.9922 .75893 
2015 Class 2 20 3.9688 .61086 
2014 Class 24 3.5208 .49955 
Total 60 3.7958 .64439 


Table 5 
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Means and Standard Deviations of Course Design Scale for Different Classes 


(Scale 1- 5: Strongly Disagree to Strongly Agree) 


N Mean Std. Deviation 
2015 Class 1 16 4.4583 .48113 
2015 Class 2 20 4.3833 51327 
2014 Class 24 3.8403 .66753 
Total 60 4.1861 .63193 


Table 6 
Comparison of Course Components in 2014 and 2015 


Individual Wiki Projects Partner Team WikiProjects 


Table 7a 
Effectiveness Means (and Standard Deviations) of Different Course Components in Helping Teachers Learn Course Content, 
Helping Teachers Learn How to Teach Course Content to Students, and Enabling Teachers to Implement Knowledge and/or 


Strategies Directly into Their Classrooms 


(Scale 1- 5: Least Effective to Most Effective) 


Effectiveness in Helping Effectiveness in Effectiveness in 
Teachers to Learn Course Helping Teachers Enabling Teachers to 
Content Learn how to Teach Implement Knowledge 
Course Content to and/or Strategies 
Students Directly into Their 
Classrooms 
Mean SD Mean SD Mean SD 
Readings 2015 CLASS 1 4.44 .629 4.19 .750 4.50 .632 
2015 CLASS 2 4.30 .979 4.05 .945 4.10 1.165 
2014 CLASS 4.08 .654 3.88 .197 3.88 .850 
Total 4.25 .773 4.02 .833 4.12 -940 
Videos 2015 CLASS 1 4.44 727 4.38 .719 4.44 .629 
2015 CLASS 2 4.45 .887 4.25 .967 4.20 951 
2014 CLASS 4.13 .680 4.08 717 3.96 751 
Total 4.32 .770 4.22 -804 4.17 -806 
Wiki Projects 2015 CLASS 1 4.38 .885 4.38 -806 4.50 .730 


[Partner Team 2015 CLASS 2 3.90 1.119 3.90 1.071 4.05 1.050 
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Projects in 2015] 2014 CLASS 3.33 1.167 3.33 1.049 3.29 1.042 
[Individual in 2014] Total 3.80 1.147 3.80 1.070 3.87 1.081 
Partner Team 2015 CLASS 1 4.06 .929 4.13 .957 4.13 885 
(2015) 2015 CLASS 2 3.55 1.276 3.55 1.234 3.65 1.182 
Group Work (2014) = 2014 CLASS 2.25 1.113 2.33 1.129 2.42 1.349 
Total 3.17 1.355 3.22 1.342 3.28 1.379 
Prior Knowledge 2015 CLASS 1 4.25 .856 4.13 .885 4.19 .834 
Thinking Prompts 2015 CLASS 2 3.75 1.020 3.75 1.118 3.80 1.152 
(2015) 2014 CLASS 3.67 1.049 3.67 1.239 3.58 1.213 
Prior Knowledge Total 3.85 1.005 3.82 1.112 3.82 1.112 
Assignments (2014) 
Module Reflections 2015 CLASS 1 4.13 -806 4.25 775 4.06 772 
(2015) 2015 CLASS 2 4.00 1.076 3.90 1.021 4.00 973 
Reflections (2014) 2014 CLASS 4.08 .776 3.96 .955 3.92 929 
Total 4.07 -880 4.02 -930 3.98 -892 
Discussion Boards 2015 CLASS 1 4.38 .957 4.38 .806 4.44 .814 
2015 CLASS 2 4.35 745 4.20 .894 4.20 834 
2014 CLASS 3.38 .970 3.25 1.113 3.38 1.056 
Total 3.97 1.008 3.87 1.081 3.93 1.023 
Presentation of 2015 CLASS 1 3.88 .957 3.88 -885 3.94 929 
Learning 2015 CLASS 2 3.50 1.147 3.45 1.191 3.55 1.234 
2014 CLASS 3.25 1.189 3.38 1.135 3.46 1.141 
Total 3.50 1.127 3.53 1.096 3.62 1.121 
Table 7b 
Means and Standard Deviations of Course Components Effectiveness Scale for Different Classes 
(Scale 1 -5: Least Effective to Most Effective) 
N Mean Std. Deviation 
2015 Class 1 16 4.2422 .63451 
2015 Class 2 20 3.9333 .68923 
2014 Class 24 3.4965 -60816 
Total 60 3.8410 .70308 
Note. The global Course Components Effectiveness scale had an alpha reliability of .95. 
Table 8 
Means and Standard Deviations of Preparation Items for Different Classes 
(Scale 1- 5: Very Poorly Prepared to Extremely Well Prepared) 
Preparation Items 2015 Class 1 2015 Class 2 2014 Class 
Mean SD Mean SD Mean SD 
Teach subject matter concepts, knowledge, and skills in 4.25 683 4.05 759 4.08 584 
ways that enable students to learn. 
Understand how different students in your classroom 4.50 .632 4.35 587 4.25 .676 
are learning. 
Set challenging and appropriate expectations of learning 4.44 629 4.10 912 3.83 761 
and performance for students. 
Help all students achieve high academic standards. 4.38 806 4.15 933 3.96 624 
Plan instruction that builds on students’ experiences, 4.50 .632 4.30 1.081 4.21 .658 
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interests, and abilities. 


problems. 


6. Evaluate curriculum materials for their usefulness and 4.19 .655 4.15 745 4.21 721 
appropriateness for your students. 

7. Use instructional strategies that promote active student 4.69 .602 4.25 .910 4.17 .816 
learning. 

8. Relate classroom learning to the real world. 4.56 727 4.35 745 4.08 717 

9. Understand how students' social, emotional, physical, 4.56 .629 4.05 1.050 4.13 .680 
and cognitive development influence learning. 

10. Identify students’ special learning needs and/or 4.13 719 4.15 .988 4.00 .659 
difficulties. 

11. Teach in ways that support English language learners. 4.50 .730 4.20 .616 4.08 717 

12. Choose teaching strategies for different instructional 4.37 719 4.20 .616 4.00 722 
purposes. 

13. Choose teaching strategies to meet different student 4.38 .619 4.25 550 4.00 722 
needs. 

14. Help students become self-motivated and self-directed. 3.94 772 4.10 852 3.71 1.160 

15. Help students learn to think critically and solve 4.13 .806 4.20 834 3.71 1.083 


16. Use multiple forms of assessment to determine 4.31 873 4.05 945 3.71 1.042 
students’ strengths and needs (e.g., observation. 
portfolios) 
17. Evaluate and reflect upon your practice to improve 4.50 .632 4.35 .671 3.92 974 
instruction. 
18. Engage in planning and problem solving with colleagues. 4.50 .730 4.45 .605 4.00 .933 
19. Use media and other technology to create learning 4.25 .683 4.30 801 3.92 929 
experiences that make the subject matter accessible and 
meaningful to students. 
Table 9 
Means and Standard Deviations of Preparation Scale for Different Classes 
(Scale 1- 5: Very Poorly Prepared to Extremely Well Prepared) 
N Mean Std. Deviation 
2015 Class 1 16 4.3717 .57317 
2015 Class 2 20 4.2105 .65848 
2014 Class 24 3.9978 .58987 
Total 60 4.1684 .61842 
The global Preparation scale had an alpha reliability of .96. 
Table 10 
Number and Percentage of Teachers in Different Classes Reporting Adequate Preparation for Each Preparation Item 
Preparation Items 2015 Class 1 2015 Class 2 2014 Class 
N % N % N % 
1. Teach subject matter concepts, knowledge, 16 100.0% 19 95.0% 24 100.0% 
and skills in ways that enable students to 
learn. 
2. Understand how different students in your 16 100.0% 20 100.0% 24 100.0% 
classroom are learning. 
3. Set challenging and appropriate expectations 16 100.0% 19 95.0% 23 95.8% 
of learning and performance for students. 
4. Help all students achieve high academic 16 100.0% 18 90.0% 24 100.0% 
standards. 
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5. Planinstruction that builds on students’ 16 100.0% 18 90.0% 24 100.0% 
experiences, interests, and abilities. 
6. Evaluate curriculum materials for their 16 100.0% 19 95.0% 23 95.8% 
usefulness and appropriateness for your 
students. 
7. Use instructional strategies that promote 16 100.0% 19 95.0% 23 95.8% 
active student learning. 
8. Relate classroom learning to the real world. 16 100.0% 19 95.0% 24 100.0% 
9. Understand how students' social, emotional, 16 100.0% 19 95.0% 24 100.0% 
physical, and cognitive development influence 
learning. 
10. Identify students’ special learning needs 16 100.0% 18 90.0% 24 100.0% 
and/or difficulties. 
11. Teach in ways that support English language 16 100.0% 20 100.0% 24 100.0% 
learners. 
12. Choose teaching strategies for different 16 100.0% 20 100.0% 24 100.0% 
instructional purposes. 
13. Choose teaching strategies to meet different 16 100.0% 20 100.0% 23 95.8% 
student needs. 
14. Help students become self-motivated and 16 100.0% 19 95.0% 21 87.5% 
self-directed. 
15. Help students learn to think critically and 16 100.0% 19 95.0% 21 87.5% 
solve problems. 
16. Use multiple forms of assessment to 15 93.8% 18 90.0% 21 87.5% 
determine students’ strengths and needs 
(e.g., observation. portfolios) 
17. Evaluate and reflect upon your practice to 16 100.0% 20 100.0% 22 91.7% 
improve instruction. 
18. Engage in planning and problem solving with 16 100.0% 20 100.0% 23 95.8% 
colleagues. 
19. Use media and other technology to create 16 100.0% 19 95.0% 23 95.8% 
learning experiences that make the subject 
matter accessible and meaningful to students. 
Table 11 
Means and Standard Deviations of Technology Use Preparation for Different Programs 
(Scale 1- 5: Very Poorly Prepared to Extremely Well Prepared) 
2015 Class 1 2015 Class 2 2014 Class 
Mean SD Mean SD Mean SD 
4.25 .683 4.30 .801 3.92 .929 


Figure 2 Self-Efficacy Scales 


Personal Teacher Efficacy [PTE] (alpha reliability = .86) 


If a student masters a new math concept quickly, this might be because | knew the necessary steps in teaching that concept. 
When the grades of my students improve it is usually because | found more effective teaching approaches. 

When | really try, | can get through to most difficult students. 

If a student did not remember information | gave in a previous lesson, | would know how to increase his/her retention in the 
next lesson. 

When a student does better than usual, many times it is because | exerted a little extra effort. 

If a student in my class becomes disruptive and noisy, | feel assured that | know some techniques to redirect him quickly. 
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If one of my students could not do a class assignment, | would be able to accurately assess whether the assignment was at the 
correct level of difficulty. 
When a student is having difficulty with an assignment, | am usually able to adjust it to his/her level. 


When a student gets a better grade than he/she usually gets, it is usually because | found better ways of reaching that student. 


General Teacher Efficacy [GTE] (alpha reliability = .71) 


A teacher is very limited in what he/she can achieve because a student’s home environment is a large influence on his/her 
achievement. 

If students are not disciplined at home, they aren’t likely to accept any discipline. 

The hours in my class have little influence on students compared to the influence of their home environment. 

The amount that a student can learn is primarily related to family background. 

The influences of a student’s home experiences can be overcome by good teaching (reverse coded). 

If parents would do more with their children, | could do more. 

Even a teacher with good teaching abilities may not reach many students. 


Table 12 
Means and Standard Deviations for Self-Efficacy 


(Scale 1- 5: Strongly Disagree to Strongly Agree) 


PTE 

Class N Mean Std. Deviation 
2015 Class 1 16 3.9861 .52411 
2015 Class 2 20 3.9611 .53745 
2014 Class 24 3.8056 .63131 
Total 60 3.9056 .57017 
GTE 

2015 Class 1 16 2.5268 .64885 
2015 Class 2 20 2.5071 .77899 
2014 Class 24 2.4881 53438 
Total 60 2.5048 .64344 
Table 13 


Correlations: Course Experience, Course Components Effectiveness, Preparation, and Teacher Efficacy 


Instructor Culture Course Component Preparation Technology PTE GTE 
Engagemen of Design Effectiveness Preparation 

t Learning 
Instructor 535" .617"* .479"* .480** .302* .250 -.183 
Engagement 1 
Culture of 1 501" .736"" 591" 537" .266" -.116 
Learning 535" 
Course Design 501** 1 .467"* .470* 344" .244 -.052 

617" 
Component 736" 467" 1 723" .618"* 389" -.063 


Effectiveness 479"* 


Preparation 
Technology 
Preparation 


PTE 


GTE 


591" 

480°" 
537" 

.302° 
.266° 

.250 
-.116 

-.183 


470" 


344" 


244 


052 


** Correlation is significant at the 0.01 level (2-tailed). 
* Correlation is significant at the 0.05 level (2-tailed). 


Table 14 


Sex Comparisons on Ratings of Course Experience, Course Components Effectiveness, Preparation, and Teacher Efficacy 


Instructor Engagement 
Culture of Learning 
Course Design 
Component 
Effectiveness 
Preparation 
Technology Preparation 


PTE 


GTE 


Table 15 


Age Comparisons (Two Age Groups) on Ratings of Course Experience, Course Components Effectiveness, Preparation, and 


Teacher Efficacy 


Instructor Engagement 
Culture of Learning 
Course Design 
Component Effectiveness 
Preparation 

Technology Preparation 
PTE 


GTE 


Sex 
Male 
Female 
Male 
Female 
Male 
Female 
Male 
Female 
Male 
Female 
Male 
Female 
Male 
Female 
Male 
Female 


Age 

30 and under 
over 30 

30 and under 
over 30 

30 and under 
over 30 

30 and under 
over 30 

30 and under 
over 30 

30 and under 
over 30 

30 and under 
over 30 

30 and under 
over 30 


N 


23 
37 
23 
37 
23 
37 
23 
37 
23 
37 
23 
37 
23 
37 
23 
37 


Mean 


30 
29 
30 
29 
30 
29 
30 
29 
30 
29 
30 
29 
30 
29 
30 
29 


723" 


.618"* 


389" 


-.063 


4.3098 
4.4493 
3.5870 
3.9257 
4.0145 
4.2928 
3.7536 
3.8953 
4.2059 
4.1451 

4.22 

4.08 
3.8551 
3.9369 
2.4348 
2.5483 


Mean 
4.4458 
4.3578 
3.7750 
3.8147 
4.1722 
4.2069 
3.8972 
3.7486 
4.1544 
4.1887 

4.20 
4.07 
3.9667 
3.8352 
2.5714 
2.3941 


Std. Deviation 


.54208 
.63905 
.68094 
.59326 
.69252 
57522 
.70888 
.70366 
.61130 
.63004 

671 

924 
.61813 
54467 
.63520 
.65337 


Std. Deviation 
.61121 
.60755 
.67706 
.63177 
.62588 
.65867 
.67058 
.72695 
.64858 
.60706 

714 
961 
58805 
.56236 
.67320 
.57734 
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Table 16 


Ethnicity Comparisons (Two Groups) on Ratings of Course Experience, Course Components Effectiveness, Preparation, and 
Teacher Efficacy 


Ethnicity N Mean Std. Deviation 
Instructor Engagement white 35 4.3143 .66447 
of color 25 4.5100 49466 
Culture of Learning white 35 3.7571 .69129 
of color 25 3.8500 58184 
Course Design white 35 4.1429 .66192 
of color 25 4.2467 59535 
Component Effectiveness white 35 3.7798 72723 
of color 25 3.9267 .67296 
Preparation white 35 4.1579 .64574 
of color 25 4.1832 59081 
Technology Preparation white 35 4.09 .887 
of color 25 4.20 .764 
PTE white 35 3.7841 55813 
of color 25 4.0756 55344 
GTE white 35 2.4898 56645 
of color 25 2.5257 .74997 


Table 17 Comparisons on Ratings of Course Experience, Course Components Effectiveness, Preparation, and Teacher Efficacy 
between Teachers with and without Summer School Teaching Experience 


Taught in Summer School N Mean Std. Deviation 
Instructor Engagement Yes 9 4.8611 .11600 
No 27 4.5324 .50071 
Culture of Learning Yes 9 4.2917 .44194 
No 27 3.8750 .70711 
Course Design Yes 9 4.4259 .54078 
No 27 4.4136 .48782 
Component Effectiveness Yes 9 3.9722 .64550 
No 27 4.1034 .69235 
Preparation Yes 9 4.4269 .81852 
No 27 4.2339 .54722 
Technology Preparation Yes 9 4.44 726 
No 27 4.22 751 
PTE Yes 9 3.7654 .53990 
No 27 4.0412 .51024 
GTE Yes 9 2.6667 .42258 


No 27 2.4656 .78805 


